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ABSTRACT 

This docuaent, sixth in a series of 11 subvoluaes ct 
a handbook prepared to provide training for educational research and 
development personnel in the deveHopaent of instructional aaterials, 
deals vith the task of developing diagnostic and evaluative tests. 
The document is organized according to the four sequential steps 
involved in performing the task. The four steps involve: (a) planning 
the development of tests; (b) developing tests for proficiency in 
criterion behavior vhich is taught in ^ach instructional unit; (ci 
developing diagnostic tests to identify the reason for failure to 
acquire the criterion behavior taught in each instructional unit; and 
(d) trying out and revising testing procedures. Bore specific 
substeps list procedures for perforaing each step. (PD) 
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FOREWORD 



This is one of a series of eleven HANDBOOK sub-volumes which 
has been prepared to provide tratnlng for educational RSO personnel 
in the development of instructional materials. 

The USER'S MANUAL, which accompanies the series, describes the 
role each volume is designed to play and the sequence recommended 
for its jse in the training process. The user 5s, therefore, urged 
to read the Instructions in the USER'S MANUAL before usinq this or 
any other separate volume. 
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TASK 

DEVELOP DIAGNOSTIC AND EVALUATIVi: TESTS 



CONVENTS 



STEPS 



Plan the development of tesL:» 



F.1J 



SUB STEPS 



Determine the size of the 
instructional unit for which a 
set of tests will be developed 



page 
F 



F.I. 2 



Hake decisions abouv the types of 
tests (and their number) In each 
set of tests which wi H be developed 
for each instructional unit 



n 



F.1.3 



Determine whether actual tests 
or storyboards for tests will be 
developed for each instructional 
un i t 



19 



f.2 



Develop tests for proficiency 
at criterion behavior which is 
taught in each instructional 
uni t 



27 



F.2.1 



Determine whether to test for 
product tind/or for process 



31 



f.2,2 



Decide on the sample of criterion 
behavior to be covered in a 
criterion test 



F.2.3 



Develop criterion test items 
or procedures 



F.2.'* 



Develop $ubcr(ter!on test Uems 
or procedures 



71 
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iv 



F.3 Develop dia<inost^c 


tests 


to identify reason 


for 


f ai lure to acqui re 


the 


criterion behavior 


taught 


in each instructional unit 



79 



Plan the use of results on 
criterion test items and on 
subcriterion test items in 
diagnosing type of learning 
failure that has occurred 



Develop diagnostic interview 
procedures to provide supple- 
mentary diagnostic information 
about results on criterion 
test items and subcriterion 
test items 



F.3. 3 Develop diagnostic test items 
designed to identify specific 
types of learning failures 



131 



F.** Try out and revise 
testing procedures 



F.'^.l 



Administer testing procedures 
to a sample of the target 
audience as a means of assess- 
ing and upgrading the mechanics 
of the testing procedures 



151 



Perform an analysis of the 
target audience by administering 
the revis«^d tests 
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STEP I r.l 



Plan the development of tests. 



F.1.1 



Determine the size of the instructional unit for which a set of 
tests will be developed. 



F.1.2 



Make decisions about the types of tests (and their number) in each 
set of tests which will be developed for each instructional vmit. 



F.1.3 



Determine whether actual tests or storyboards for tests will be 
developed for each instructional unit. 
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STEP 



OVERVIEW 



INfH;T 



ACTION 



Statement of objectives 
simulation decisions 



Plan the number and 
types of tests to be 
developed 



OUTPUT 



DECISIONS 
about tests to be 
developed 



Statements of 
criterion ob.lectives 
on FORM D.2(l), 
grouped by lessons 



Stateuent of 
subcrlterion objectives 

on FORl'I D.2(l), 
grouped by 
criterion objectives 

iv 



Determine the number 
of lessons to consider 
in making plans for 
development of tests 

ii 



Determine the niunber 
of subcrlterion tests 
needed per set of tests 
devc^ed to an 
instructional unit 



DECISION: 
-LESS than a lesson, 
-A lesson, or 
-MOKE than a lesoon 



as a testing unit 



iii 



DECISION: 
As many subcrlterion 
tests per criterion 
objective (the 
instructional unit) as 
there are subcrlterion 
objectives y1 



Criterion behavior in 
each identified 
Instructional unit 
F.1.3| ♦ 

sisulation decision 
(Step E) for it 

vii 



Assess cost of prepar- 
ing and tryin^^ out 
test of criterion 
behavior using required 
raed5a; 

decide whether costs. . . 

. . _ viii 
are aggeptablg 



DECISION 
to develop: 

-Actual tests 
OR 

-Storyboard for tests 
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STEP I F.I 



PAGE INDEX 

CRtTERIA FOR 





IDENTIFYING INPUTS 


ACTION TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 


F.1.1 


-MATRIX: Unit 
of instructional 
material to be 
tested 8 


-MATRIX: How 
many tests 
to develop per 




SUMMARY OF 

PROCEDURES , . . 9 



-MATRIX: Three 




-MATRIX: How many 




SUMMARY OF 




types of 




types of test to 




PROCEDURES . . 


. 17 




.]k 


prepare .... 15 













-MATRiX: When to 




SUMMARY OF 




F.1.3 




develop actual 
tests vs. 

storyboard ... 2? 




PROCEDURES . 
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PREVIEW OF THE NEXT SobSTEP 



YOUR PRODUCT 


Plana to develop testing units fo:* each criterion 
behavior (reaultina in lesson units having as 
many testing units cs criterion behaviors covered in 
them). 


WHAT YOU WILL 
WORK FROM 


(1) Statements of criterion objectives grouped by 
lesson units. 


WHAT YOU WILL 
DO 


(1) Determine the number of criterion objectives 
covered In a lesson unit. 

(2) Decide whether the testing unit wilT be less than 
a lesson unit, a lesson unit, or two or more 
lesson units. 


FORMS YOU WILL 
USE 


None 
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DESCRIPTION OF Sub STEP 



INPUT 



Statements of 
c riterion objectives 
on FORM D*c?(l) 
f^rouped by lessons 



F.1.1 



ACTION 



Determine the nunber 
of lessons to consider 
in makinc plans for 
development of tests 



ii 



OUTPUl 



DKCISION: 
-LESS than a Icnson, 
-A lesson, or 
-MORE than a lesson 

as a TESTING unit 



ill 



Job Aid ContenU 



CRITERIA fO'A 



IDENTIFYING INPUTS 


ACTrON TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 


-HATRIX: Unit 


-MATRIX: How 




SUMMARY OF 


of Instructional 


many tests 




PROCEDURES ... 9 


material to be 


to develop per 
















R(*(juir(M} 


Main uils 





COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 






FORM D.2(l) 

carried forward from 


E.I 
E.2 
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JOB DIAGRAM 



INPUT 



Statements of 
criterion objectives 
on FORJ'I D.2(l) 
grouped by lessons 



ACTION 



Determine the nuTiber 
of lessons to consider 
In making: plans for 
development of tests 



il 



OUTPUT 



DECISION: 
-L'ilSS than a lesson » 
A lesson, or 
MORE than a lesson 

as a TESTING unit 



ill 



TESTING PLANS 



A s ingle 


lesson covers 


more than 


one 


or 1 tcr ion 


ob jcct i ve 




l.a 



Treat instruct ion (in 
a lesson) covering each 
criterion objective 
as a uni t for testing 



11.3 





As many tests 1 




per lesson 1 




as there are 1 




criterion objectives 1 


► 









A single lesson covers 
only one 

criterion objective 



i.b 



Treat the entire lesson 
as the unit for testing 



i i.b 



Only one test 
per lesson 



Several lessons cover 
only one 

cri terion objective 



l.c 



Treat all the lessons 
required to cover 
a criterion objective 
3S the unit for testing 



i i .c 



Only^ 



one test 

the lessons 



for all 

rcqui red to cover 
the criterion objective! 
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JOB PROCEDURES 





page 


Size of instructional unit to use 
as a basis for test development 


8 


How many tests to develop per lesson 


8 


SUMMARY OF JOB PROCEDURES 


9 
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F.1.1 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING APPROPRIATE INSTRUCTIONAL UNIT SIZE 
A5 A BASIS FOR TEST DEVELOPMENT 



APPROPRIATENESS 
OF UNIT 


APPROPRIATE 
Instruct ional Mni t 
to be tested 


INAf^PKOPRIATE 
Instructional Uii t 
to be tested 


CRITERIA 


All the instructional nntevial 
related to 

each CRIrtsRlON objective 

can range from: 

-less than a single lesson, to 
-multiple lessons 


Instructional mater-lal 
re lated to 

less than or part of 
a criterion objective 



F.1.1 


DETERMINING HOW MANY TESTS 
TO DEVELOP PER LESSON 


DECISION 
MATRIX 








CONDITIONS 


A single lesson 
covers more than 
one criterion objective 


A single lesson 
covers only one 
criterion objective 


It takes several lessons 
to cover a single 
criterion objective j 


ACTJOf* 
TO TAKE 


Tt*eat instruction 
covering each 
criterion objective 
as the unit to be 
tested 


Treat instruction 
covering the one 
criterion objective 
as the unit to be 
tes ted 


Treat instruction 
covering only a 
complete critc^^ion 
objective as the unit 
to be tested 




Develop as many tests 
per lesson as there are 
criterion objectives 


Develop only one set 
of tests for that 
lesson 


Develop ojily one set 
of tePitfi for all the 
lessons it takes to 
cover the criterion 
objective 
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F.1.1 



ILLUSTRATION SUMMARIZSNG PROCEDURES FOR DETERMINING HOW MANY 
INSTRUCTIONAL UNITS THERE ARE AND HOW MANY TESTS WILL BE NEEDED 



n 

Inspect the collection 
of '*ob ject i ves" forms 
[FORM D.2{1)] grouped 
by lessons 

Identify the number of 
cri ter ion objectives 
covered by each lesson: 

-None 
-Only one 
-More than one 



#2 

Make plans to develop 
a set of tests for each 
cri ter ion objective 
identified 

Make plans to have as 
many sets of tests 
devoted to a lesson 
as there are criterion 
objectives : 

-None 
-Only one 
-More than one 



FORM D.2(I) 



PREVIEW OF THE NEXT SubSTEP 



YOUP PRODUCT 


Piano to develop a sub-cr*iterion test for each 
sub-criterion objective. 


WHAT YOU WILL 
WORK FROM 


(1) Statements of sub-criterion objectives; all those 
relevant to a given criterion behavior grouped 
together* 


WHAT YOU WILL 
DO 


(1) Determine how many sub-criterion tests will be 
prepared for each Instructional unit. 


FORMS YOU WILL 
USE 


Hone 
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DESCRIPTION Or Sub STEP 



F.K2 



INPUT 



Statement of 
sub-criterion objec- 
tives on FORM D.2(l) 
grouped by criterion 
objectives 



ACTION 



Determine the number 
of sub-criterion tests 
needed per set of tests 
devoted to on 
instructional unit 



OUTPUT 



DECISION: 
As many sub-criterion 
tests per criterion 
objective (the ins true- 
tioi.al unit) as there 
are sub-critrrion 
objectives 



Job Aid Contents 



CRITERIA FOR 
IDENTIFYING INPUTS 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



■MATRIX: Three 
types of 
test 



1^ 



'MATRIX: How 
many types of 
test to 
prepare . • . 



15 



SUMMARY OF 
PROCEDURES 



17 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Identification of 
instructional uni t 


F.I.I 


FORM D.2(l) 

carried forward from 


F.l.l 
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Sut.- STEP 



F.1.2 



JOB DIAGRAM 



INPUl 



:;tntenent of 
uub-critcrion objec- 
tive r. on FOIC! D.2(l 
^^ rou: ed by criterion 
objectives 



IV 



ACTiON 



Determine the number 
of sub-criterion tests 
needed per set of test: 
devoted to an 
instructional unit 



OUTPUT 



I DKCISIOII: 
As many sub-criterion 
tests por criterion 
objective (the instruc- 
tioK;il unit) ^.s there 
are su^-criterion 
objective's 



JO, 



on FORM D.2(I) 



Only J cr i ter i on 
object i ve recorded 



Plan on development 
only of a cri terion 
test 



V. a 



DECISION 



A cri terion test 
to be developed 



VI .a 



Associated with a 
cri terion objective: 

-SubcrJ terJon 
objective (s) recorded 

-No 
preparatory 
objective (s) recorded 



i v»b 



In addition to above: 

Plan on development of 
a subcr! terion test to 
cover 

subcrlterlon objectives 



-A cri terion test 
to be dcve loped 
+ 

-As many subcriterion 
tests as there are 
subcr f terion 
object i ves 



Associated wi th a 
cri ter i on objective: 

-Subcrlterlon 

object ivc(s) recorded 

•Preparatory 

Object ive{s) recorded 



I v»c 



In addition to above: 

Plan on development of 
a preparatory test to 
cover 

preparatory objectives 



v.c 



-A cri terion test 
to be developed 
+ 

-As many subcriterion 
tests as there are 
subcr I terion 
object i ves 

+ 

-As many preparatory 
tests as there arc 

preparatory objectives 
, V i ■ r. 
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JOB PROCEDURES 




page 


Types of tests to develop 
for e.ich i r-> \ r ;, t ional un i t 




How many ''prcporntory" tests 
to develop per set of tests 
for each instructional unit 


15 


SUMriA,"^v or FROCLDUr.rS 


17 
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F 1 2 


CRITERIA FOR IDENTIFYING THE TYPES OF TESTS 
TO fiF nrVELOPED FOR FACH 1 NSTRUCT 1 QNAI UNIT 


IDENTIFICATION 
MATRIX 








TYPES 
OF TESTS 


CRITERION 
TEST 


SUBCRITERIOM 
TEST 


PREPARATORY 
TEST 


CRITERIA 


A teat relevant to the 
criterion objective 
covered in the 
instructional unit 


A test covering each 
and every 

aubcnterion objective 
associated with the 
criterion objective: 


A test covering all 
preparatory 

objectives ( tec ting 
for corrrponent skills) 


EXAMPLES 


Criterion Behavior 


The dXiXt^on ta>t 


A subcriterion teat 


A tut 0|( compontnt 


is to compute the 
size of the 
buoyant force 
exerted on a 
submerged object 
(by subncrging 
the object and 
weighing the 
ove rf I ow) 


maid KcqiuAt tixa 
Atiidant to CKltibit tlit 

{m ^t£ittd to tilt mti 


miglit call ijo^ tiiz 
i^txidtnt to htaXt tJit 

tilt i^tight oi tilt 

OVCA^^CK'j Md tilt 

mcu))vitudt 0^ tilt 
buoifOiit f^o^ct (i.e., 
iroduce a formula) 


6lUt(!J> fnigkt call ^ox 
tilt 6tiidt^tt. iqivta an 
OAACKO KtpKC^ tilting an 
apico^cf ijo^ce and an 
aviO(0 /Ltp^Uentcng a 
damooAd ^oKct] to 
htltcX tilt owt KtpKt- 
JbtntJing tlit '^baotjant" 





\ 
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F.1.2 

DECISION 
MATRIX 



DETERMINING HOW MANY PREPARATORY TESTS TO PREPARE 
PER SET OF TESTS DEVOTED TO EACH INSTRUCTIONAL UNIT 



CONDITIONS 



-NO subcriterion or 
preparatory objectives 
have been prepared 
relating to a 
criterion objective 



ACTJON 
TO TAKE 



-Subcri terion 
objectives have been 
prepared relating to 
a criterion objective 
AND 

-NO 

preparatory objectives 
have been prepared 
relating to a 
criterion objective 
(or to a subcrtterlon 
object I ve) 



•Preparatory 

object i ves (cover i nti 
component skM Is) 
have been prepared 
relating to: 

••A criterion 
object Ivc » or to a 

••Subcriterion 
objectives relating 
to the criterion 
objective 



'Plan to include in 
the set of testo: 

•*Only a criterion 
test 



IN ADDITION TO: 
A CRITEmON TEST 

-Plan to include in 
the set of tests: 

"Only a subcriterion 
test 

'Plan as many such 
tests as there are 
suhcriteriart ohjec 



IN ADDITION TO: 
A CRITERION TEST 

AND 

SmCRITERION TESTS 

-Plan to include in 
the set of tests: 

Preparatory 
tests 

-Plan as many such 
tests as there are 
preparatory 
objectives 
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FJ.2 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED 
IN DETERMINING HOW MANY OF EACH TYPE OF TEST TO INCLUDE 
IN EACH SET OF TESTS PER INSTRUCTIONAL UNIT 



#1 



#2 



Inspect the coMccticn 
of *'objccti ves*' forms 
IFORM D.2(l)] associated 
with each criterion 
objective 

Identify the ni.»iber of 
sub-objectives 
associated with each 
criterion objective 



Make plans to develop 
a test for each 
sub*obJect ive 

••Subcriterlon objective 
••Preparatory objective 



FORM D.2{l) 



«1LJM*i1 Witt 



erJc" 
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PREVIEW OF THE NEXT SubSTEP 



YOJR PRODUCT 


A decision to develop tests using media required by 
criterion behavior or to use ctoryhoard teats because 
they are cheaper. 


WHAT YOU WILL 
WORK FROM 


(1) Classification of the criterion behavior to be 
practiced In an Instructional unit. 

(2) Decisions which specify that a sinulatlon of the 
criterion behavior will be practiced. 


WHAT YOU WILL 
DO 


(1) Assess the costs of developing .ind uslnq tests 
using the media needed for testing criterion 

K/af-j *i V/ 1 nr* nr A cimul.ilpd version of it. 


FORMS YOU WILL 
USE 


tJonc 
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DESCRIPTION OF Sub STEP 



F.K3 



INPUT 



Criterion behavior 
in each identified 
instructional unit 

simulation decisions 
(Step E) for it 



vii 



ACTION 



Assess cost of prepar- 
ing and trying out test 
of criterion behavior 
usinr: required media; 

decide whether costs 
are acceptable 

viii 



OUTPUT 



DECISION 
to develop: 

•Actual tests 
OR 

•Stoiryboard for tests 



Ix 



CRITERIA FOR 
IDENTlFVIfWG INPUTS 



Job Aui C\>ntenls 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 





-MATRIX: When 
to develop 
actual tests vs. 
storyboard , . , 22 




SUMMARY OF 
PROCEDURES ... 23 



Ho<^uirr(l Matrrials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


!dent i f i cat ion of 
instructional unit 


F.l . 1 


FORM EJ(I) 

S i mu ) a t i on decision 


E.2.3 




Ident if ication of 
types of tests :o use 
,fnr iini t 


F, 1.2 
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Sub- STEf I FJ,3 I 



JOB DIAGRAM 



INPUT 



Criterion behavior 
in each identified 
instructional unit 

sinulation decisions 
(Step K) for it 



VI 1 



ACTION 



Assess cost of prepar- 
ing^ and tryi out test 
of criterion behavior 
ucinr> requi -ed media; 

decide whether costs 
are acceptable 

viii 



OUTPUT 



DECISI0I3 
to develop: 

-Actual tests 
OR 

-Stor^'board for tests 



Cost of using required 
media to test criterion 
behavior or simulation 
of it 

is expected to be LOW 



VM .a 



Plan to use required 
media and to develop 
actual tests 



vill.a 



DECISION 



To develop and use 
non*cost1y media in 
actual test 



•Papefand^pencl 1 

•Slides 

* Audio tape 

•Inexpensive objects 



Cost of using required 
media to test criterion 
behavior or simulation 
of it 

fs expected to be HIGH 



vii.b 



Plan to use required 
media and to develop 
storyboard for tests 



vl I i .K 



To develop and use 
s toryboa rd version 
of test 



during tryout of test 
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JOB PROCEDURES 





page 


Determining whether to develop actual 
tests or storyboards for tests 


22 


SUMMARY OF PROCEDURES 


23 


* 
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F.1.3 


DETERMINING WHETHER TO DEVELOP EITHER 
ACTUAL TESTS AND/OR ST0RY80ARDS FOR TESTS 


DECISION 
MATRIX 






CONDITIONS 


-Preparing a test of criterion 
DeridV 1 or 

OR 

-Preparing a test of simulated 
cr f ter i on behav ior 


•Preparing a test of criterion 
Denav Bor 

OR 

-Preparing a test of simulated 
cri terion behavior 




results in LOW costs 


results in HIGH costs 




due to combinations of: 


due to combinations of: 




• •Time 
• 'Money 
• -Faci 1 i ties 
• •Personnel 
• •Materi als 


• •Time 
• •Honey 
• •Faci It ties 
• •Personnel 
• •Materials 


ACT r ON 


Plan to develop 
an ACTUAL TEST 


Plan to develop 
test in STORYBOARD F0RM^4 



*Follcmng try out of the adequacy of a test in storyboard form, the actual 
test would be developed. 

^Subsequent sections do not provide details of procedures for developing and 
trying out materials in storyboard form. 



EXAMPLES 



■Use o 
medi a 
aud I o 
etc* ) 



f relatively i nexpens f ve 
(paper and pencil, si ides » 
tape, inexpensive objects, 

tilt cKiXoAA^on be^havlon. i6 
"the, Idejiti^lcatlon ttjpu 
Oj{ tcav2^*" StLdu ol did 
tcave/^ to b<L tdrntilto^d a/it 
pn,tpaA^d, 6lnc(L It aj> not 
co^tZtf to do 

thcAt tikoJiif to be a neerf 
to ^Q.vu>2. tha ^*6ll0t&" 0^ 
the, Izave^ 



Use of costly media (film, TV, 
complex s imulators , etc.) 



e.g., cjUtoA^ion bdiavlon, (fat a. 
dtvQjLopoA o/( AjUtAuctionat 
fttoteA^af^ tn\JotvQ/> ^^p^apa/UL" 
tlon 0|( tAjcUninq ^xjbrUi**; 
a toMt 0^ thU be,kavlon, 
aAX,ng actual fi<Jbn coaid 6c 
C06tlij; tkoAQ^ioKQ,, the ^Itm 
t^\t ^ f^AJUit 6cAxptcd land 
tfuexl oat that ^o\m] 
bej{oA.e tt JU acXiiaUjLf 
ptoducQjd 
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ILLUSTRATION SUHMARIZING PROCEDURES FOR OElTERHINING 
WHETHER, FOR EACH INSTRUCTIONAL UNIT, 
tq nPVPi np actuai tp<;ts qr ^Tnnv aoARD for tfsts 



#1 


n 




FOR EACH CRITERION OBJECTIVE 










(which defines the Instruc- 










tional uni t) 










St 1 n C nf* f f t i^m ff\ 1 Ia^^ inn 
a* iiiapcww I. nc CO 1 ICwl'vTl 




a. Assess the costs of 






of "objectives'* forms 




using the media 






[FORM 0.2(1)) 




required for an actual 






associated wi th that 




test of the criterion 






cr 1 terlon objective 




behavior (or the 
simulated version 
decided on) 






b. Inspect the simulation 




b. Plan the use of actual 






decision [on FORM E. 1 (1)) 




tests or storyboard 






made with respect to the 




version during tryout 






criterion behavior 




of test based on cost 






covered in the statement 




considerations 






of the objective 













ST6F I F.I 



COMPLETION CHECKLIST 



IDENTIFIED 


PERFORMED 


PRODUCED 


FORMS COMPLETED 


The number of 
instruct •onat units 
for which tests 
be needed 










The number of 
different types of 
tests that wi 11 be 
necessary for 
instructional uni t 











Decided whether to 
develop actual 
tests ur a 
storyboard for 
tests 
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STEP 



F.2 



F.2 



Develop tests for proficiency at critfe'ion behavior which is taught in each 
instructional unit. 



F.2.1 



Determine whether tc test for produc t and/or for process , 



F.2, 2 



Decide on the sample of criterion behavior to be covered in a 
criterion test. 



F.2. 3 



Develop criterion test items or procediires. 



F.2, 4 



Develop subcriterion test items or procedures. 
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STEP 



F,2 



INPUT 



All analysis results 
for each cntericn 
behavior ^ 

statement of objectives 
+ 

simulation decisions 
testing decisions 



OVERVIEW 



ACT(ON 



Develop tests of 
proficiency for each 
criterion behavior 



OUTPUT 



Criterion tests 
+ 

)ubcriterion tests 
assessing proficiency 
in criterion behavior 
and associated 
Jubcriterion behaviors 



F.2.1 



F.Z.2 



F.2.3 



F.2,4 



Grouped statement of 
objectives 

4 

associated learning; 
analysis 



Statements of 
objectives 

task analysis forms for 
each criterion behavior 

iv 



Statement of 
criterion objective 

sampling decision 

simulation decision 



Vll 



Statement of 
subcriterion objective 

sajnpling decision 

simulation decision _ 



Decide whether to 
test for criterion or 
su'ycriterion outputs 



ii 



Determine hov much 
of criterion behavior 
to sample on test 



Prepare criterion 
test item(s) 



viii 



Prepare subcriterion 
test Item(s) 



xi 



DECISION 
to test for 

-Criterion output 
(product) 

-Subcriterion output 
(process) iii 



DECISION: 

-Number of INPUT 
classes to sample 

-Number of test items 
per class to develop 

vi 



CRITERION TEST item(s) 
containing: 

-GIVEN 

-STUDENT WILL 

-RESULTING IN . 

ix 



Subcriterion test 
items containing: 

-GIVEN 

^STUDENT WILL 
-^RESULTING IN 



xil 
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STEP I F.2 



PAGE INDEX 



CRITERIA FOR 





IDENTIFYING INPUTS 


ACTION TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 


F.2.1 


-MATRIX: What is 

"product" and . 

"proceiS" .... 3^ 
-HATRIX: Three 

types of 

Outputs to be 

tested 35 

-MATRIX: Functions 

served by ^ 


-MATRIX: When 
to test for 
cri terion and 
for preparatory 
outputs .... 38 




SUMMARY OF 
PROCEDURES . . 39 



F.2. 3 



•MATRrX: Two 
requi red sampl ing 
decisions ... 44 


-MATRIX: How many 
INPUT classes to 
sample .... 45 

-MATRIX: How many 
test items per 
class to 

develop .... 46 




SUMMARY OF 
PROCEDURES . . 47 




-MATRIX; Types of 
test i terris . . 51-57 


-MATRIX; Making 
scoring 

"objective" . 59"6l 
-MATRIX: What to 
include in i terns 
and where to get 
information . . 67 


-MATRIX: Adequacy 
of test 


FORM F.2(I) . . 65 

SUMMARY OF 
PROCEDURES . . 68 





-MATRIX: 






FORM f.2{]) 




F.2. 4 


Difference between 
crl terion and 
subcrlterton test 






SUMMARY OF 










PROCEDURES . 


. 75 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


Plans to develop teste uhich j.ir.esa proficiency at 
cTitf-ri' nn hph/rviop and. whcn riecesBCTii, VT'oficicncu at 
intermediate goats. 


WHAT YOU WILL 
WORK FROM 


(1) Grouped statements of objectives. 

(2) Grouped analyses of criterion behavior. 


WHAT YOU WILL 
DO 


(1) Identify anticipated length of instructional 
unl ts 

(2) Based on anticipated lesson stz0>| deterinlne 
whether to prepare tests on criterion bv'^havfor 
and/or on criterion progress. 


FORMS YOU WILL 
USE 


None 



ERIC 



DESCRIPTION OF Sub STEP 



F,2J 



INPUT 



Grouped statement of 
objectives 

associated learning; 
analysis 



AC: ION 



Decide whether to 
test for criterion or 
subcriterion outp\.ts 



OUTPUT 



DECISION 
to test for 

•-Criterion output 
(product ) 

-Subcriterion output 
(process) 



iii 



CRITERIA FOR 
fDENTIFVING INPUTS 



Job Aid Contents 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: What is 




-MATRIX: When 




SUMMARY OF 




•'product*^ anu 




to test for 




PROCEDURES , , 


• 39 


'^process" , . , , 


cr 1 ter ion and 








-MATRIX: Three 




for subcri terlon 








types of 












outputs to be 














35 










-MATRIX: Functions 












served by 














38 











Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Test planning 
deci sions 


F.I,3 


Col lection of forms 
carried forward from 


F.1.3 























































3J 



Sub- STEP i 



JOB DIAGRAfl 



INPUT 



ACTION 



Grouped statement of 
objectives 

associated learning 
analysis 



Decide whether to 
test for criterion or 
subcriterion outputs 



ii 



OUTPUT 



DECISION 
to test for 

-Criterion output 
(product ) 

-Subcriterion output 
(process) 



Statement of objectives 
identifies a 
CRITERION behavior 



I .a 



Plan development of a 
test of proficiency 
at criterion behavior 



li.a 



CRITERION test of 
prof i clcfjcy (measures 
cri terion output) 



Mi. 4 



Statement of objectives 
ident i f ies 

subcriterion behaviors 
+ 

instructional unit 
covering cri terion 
behavior is LONG 



l.b 



Plan development of a, 
test of proficiency 
at subcrl terion behavloh 



1Kb 



SUBCRITERION TEST 

•As an interim test 
of proficiency 
AND 

*As a diagnostic test 
(measures subcriterion 
outputs) 



lli.b 



Statement of objectives 
Ident I f j es 

subcriterion behaviors 

instructional uni t 
covering criterion 
behavior is SHORT 



Plan development of a 
test of proficiency 
at subcriterion behavlo 



li-c 



PREPARATORY test 

-As a diagnostic test 
(measures subcriterion 
outputs) 



ill.c 
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BACKGROUND INFORMATION 





page 


Distinction between 
"product" and "process" 




Three types of outputs 


35 
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CRITLkIA for DISTINGUISHING BLTWLlN PRUDUCT AND PROCESS 


IDENTIFICATION 
MATRIX 








r KUL t 


CRITERIA 


A product ic defined as the f inal 
OUTPUT of a criterion behcanor 


ProccGc 13 defined by arui mraQured 
by the OUTPUTS or products of 
unmodified prepcD\2toru bchaviora 

J?^i/*?T IJVr/'i' 7jT> ^Ify^ /^Y*^ f'/^l^'J /IM 

b^havir^ 


IMPLICATIONS 


The testinq of bchavfor Is therefor 
or products; but outputs may be at 
In effect, t'^stlno for process meaa 
partial or preliminary 


e always made in terms of outputs 
differing stapes of completion, 
s to test products which are 

1 




EXAMPLES 


MATHEMATICS: addlw 

e.g., a *'6um'* >c4 tJic i^jiat oatmt 
0(5 addinc^ mitribcAS and ccui be 
u4cd to ta>t {^OK did 

ENGLISH: iOU^ 

e.g., a completed c^^M ^ tlxa 

ILnat Oittpiit ox pioducX imd 
can be u^cd to a^6c66 tJic 
c^uXoAAon bcJiavioA lo^iXtoig 
an e^6at^ 


1 

g tjcc-di^it jTumbe^ 

e.g., tire an^iocx in tlie dinit^ 

iontif) cotiunn wutrf be the 
autpiU 0(5 the pKepoAatoxu 
bdiavAjox and can bt u&ed 
to toji^t iox tJie p^oeth^ 
oi "arfctuig" 

tcnq cji e^bojf 

e.g., an otit6c«e coatd be Mcd 04 
tlie oatpj^t 0^ a pAepoAotoM/ 
bdiavioA and ubed to a^^us 
ox tejbt the £XoceA6 o^ 
"MXiXtuig an eA6aij^ 





i 




F,2,l 

IDENTIFICATION 
MATRIX 



CRITERIA rOR IDENTIFYING THREE TYPES OF OUTPUTS— Fl NAL OR PREPARATORY 



TYPES OF 
OUTPUTS 


1 


II 


III 


CRITERIA 


Outputs cu-c: 

— Tnvtn ^ hi ^ 

-Observable 

-Permanent 

• Obje eta J th i njs 
*h'ritten words or 
sifnhols 


Outputs are: 

-Observable 
-Transient 
•Motor behavior 


Outputs are: 

-Cognitive 

-Vnobservable 

-Transient 

• Identif icatio},J 

•Thoughts 

•Hcadina 


IMPLICATIONS 
FOR TESTING 


1 II 1 » type v/i vUlPUl 

can t>e measured: 
-Direct ly 
-At any time 


To be rneasyred^ this 
type of output : 

-Has to be measured 
at the time it occurs 

-Has to be recorded for 
subsequent measurement 


This type of output 
has to be measured: 

-Indirectly 

-By requi ring the 
student to do 
something else 



c,ri,, t}it product ^ 

c, q» , a dianicat 
dl6tiltaX(L 
{output c(y an 
ZKptKmznt) 

ii.g* , an Q^6a(/ 



e,fl- f ^pcec/t (can be 

^c.ton.d^d on tapt 
OA a pcAmantktC 
Ktcjond) 

e.g., oMitctic outcomes 
{mov2jr^znt6 oA 
ott(pai& OK A.c^u>fti 
Oft movcjn(Liit6 as 
outputs) 

•A dive. 

•Hex^ght of^ a pcla 
vaatt icun bt 
KtCOKdad on 

l*g*, iUitig iivitAwnavU 

•HoiV held OK 
movzd 



e,^}., Kti^ding [to bt 

tcste.d^ otto^tions 
have, to be asktd 
about i^kat vjos 

t*Q,, <dtiitif^<.ait>ionA 
{to bt tested, 
the student has 
to be, KJtqwiAe.d to 
point to Aame- 
tliing OK to labtl 
someXlUng] 
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JOB PROCEDURES 





page 


Functions served by testing 


38 


Determining types of tests to develop 


38 


SUMMARY OF PROCEDURES 


39 




F.2.1 


CRITERIA IDENTIFYINC THE RJ/'.nOttS SERVED BY TESTING: 
FOR CRITERION OUTPUTS AND FOR SIBCRITERION OUTPUTS 


IDENTIFICATION 
MATRIX 






TYPE OF 
TF5T 


Test of 
CRITERION outputs 


Test of 
StiQCRITERION outputs 


CRITERIA 


'Test of student proficiency at 
criterion behavior 


-Test of student proficiency at 
si^cr^^^riofi behaviors 


FUNCTIONS SERVED 


Assess the adequacy of the total 
instructional unit to produce a 
predetermined level of student 
prof i ciency in cr i ter * on bchav ior 


Assess the adequacy of a portion of 
the instructional unit to produce 
a predetermined level of student 
proficJency in a subcriterJoo 
behavior: 

••Can be used to measure Interim 
progress ; 

AND/OR 

••Can hi^ used di agnost i cal ly to 
recor^l weaknesses in parts of 
instnKtional methods 








F.2.1 


DETERHIWjNG \VHEN TO TEST FOR: 
CRITERION AND SUBCRlTERIOi OUTPUTS 


DECISION 
MATRIX 






CONDITIONS 


For ANY criterion output 


-When ^mtructional materials for 
a givaii criterion behavior are 
spread over several lessons 
ANO/OR 

-When it Is difficult to acquire 
the criterion behavior 


ACTION 
TO TAKE 


Plan to develop on inatrument 
to t3st for proficiency of 
CRITERION behavior 


Plan to develop an instrment 
to test for: 

— InterSm progress; or for 

— Diagnostic evidence of learning 
diffiailties i 
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1 LLUSTRATION 


SUMMARIZING PROCEDURES INVOLVED IN DECIDING 






WHEN TO 


TEST FOR "PRODUlT*' (CRITERION OUTPUT) 




F.2,1 


AND 


FOR "PROCESS" (PREPARATORY OUTPUT) 





#» 




#2 




a. Inspect the decision about 
size of instructional unit 
made In Sub-STEP F. K 1 




a. 


Plan to develop a criterion 
test for proficiency at the 
cr i ter Ion behavior 






b. Inspect the collection of 
analysis forms and state- 
ments of objectives for 
determining anticipaLcd 
number of lessons likely 
to be required to teach a 
given criterion behavior. 




b.l. 
b.2. 


For LONG instructional 
units, plan to use prepara- 
tory tests (associated with 
unmodified preparatory 
objectives) as: 

' • Interim tests of 
prof i cicncy 

••Diagnostic tests 

For SHORT instructional 
units, plan to use prepara- 
tory tests (associated wi th 
unmodified preparatory 
objectives) as: 

••Diagnostic tests (only) 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A dcciaion as to hao many of the INPUT classes or 
ACTION classes to sample on a test as well as ha) 
many test items fov each class to develop* 


WHAT YOU WILL 
WORK FROM 


(1) Statements of objectives 

(2) Task analyses For each criterion behavior. 


WHAT YOU WILL 
DO 


(1) Determine how much of the criterion behavior to 
test for in o test. 


FORMS YOU WILL 
USE 


Hone 




40 



DESCRIPTION OF Sub-STEP 



INPUT 



Statements of objectives 
+ 

task analysis Toms for 
each criterion behavior 



IV 



F,2.2 



ACTION 



Deterniue how nuch 
of criterion behavior 
to sample on test 



Job Aid Contents 



CRITERIA FOR 
IDENTIFYING INPUTS 

-MATRIX: Two 
requ r red sampi ? ng 
decisions ... 



ACTION TO BE TAK5N 



-MATRIX: How many 
INPUT classes to 
sample ^5 

-MATRIX: How many 
test items per 
class to 

develop . • • • A6 



STANDARD FOR OUTPUTS 



OUTPUT 

DECISIO:;: 

-Number of INPUT 
classes to sample 

-Number of test items 
per class to develop 

vi 



FORMS TO USE 

SUMMARY OF 
PROCEDURES . . . 



Required MaleriiUs 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Decision to test for 
product/process 


F-2.^ 


Collected forms for 
cri terion behavior 
carried fon-zarH from 


F.2.I 
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Sub- STEP 



JOB DIAGRAM 



INPUT 



Statement of objectives 

task analysis ferns for 
each criterion behavior 



iv 



ACTION 



Determine how nuch 
of criterion behavior 
to sample on test 



OUTPUT 



DECISION: 

-Number of INPUT 
classes to sojnple 

-Nunber of test items 
per class to develop 

vi 



STATEMENT OF OBJECTIVES 
I dent i f ICS 



•Relat i vely more 
complex performance: 

••There are many 
classes^ and/or 

••There are many 
members per class 

AND: 

•Performance i nvol ves 
TRANSFER vbecause of 
above conditions) iv»a 



Plan to sample 
INPUT classes 
less heavi ly 



v.a 



PLANS 



■ Al 1 classes to be 
sampled 

BUT 

■ Fev?er members of 
classes of INPUTS 
to be sampled 

* ' Fewe r test items 
per class to be 



sampled 



v\ .a 



•Relatively less 
complex performance: 

••There are few 
classes, and/or 

••There are few 
members per class 

AND 

Pe r f o rmance i nvo I ves 
RECALL (because of 
above conditions) iv»b 



Plan to sample 
INPUT classes 
more heavi ly 



v.b 



' All classes to be 
sampled 

AND 

-More members of 
classes of INPUTS 
to be sampled 

• • More test i terns 
class to be 



per 
sampled 



vl.b 




k2 



JOO PROCEDURES 





page 


What and how rr.uch of 
criterion behavior to sample 




Determining the sampling of 
classes of inputs 




Determining how many test 
items to be developed 
per class of inputs to be 
sampled 




SUMMARY OF PROCEDURES 


^7 



*This section provides rules of thumb only. Thmj are based on 
teaming considerations primarily and testing" coixsiderations 
secondarily. Consult other works on '^test development'^ con^ 
ceming testing issues related to test length. 
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F.2.2 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING TWO REQUIRED TYPES OF SAMPLING OECISIDHS 



TYPES OF 
DECISIONS 


WHAT 

elements of cn'tcrton behavior 
need to be sampled on a test? 


HOW MANY 
test items should there be 
for each element decided on? 


CKITEKIA 


l-rnen they are identified in 
statencnts of objectivcoy oliould 
you Simple: 

'All classes (HJPUTS or ACTIONS) 
or just sone of them 

'All members within each class, 
or just come of then 


WJien it is decided how many 
classes and hoj many members 
irithin a class are to be caroled, 
should you: 

'Develop one or more test items 
per class 

'Develop one or more test items 
per member vithin a al/zss 



iNPiir 



ENGLISH 
ACTION 



am 



it 



On a criterion test should each 
of the three INPUT classes be 
represented; should all three 
members of the class of third 
person pronouns be represented? 



On a criterion test how many test 
items should there be for each of 
the three classes; how many for 
each member within the class of 
third person pronouns? 



ERIC 



ui 



o 

O 



> 

3 



a. 



o 

UJ 

«/) 



ft. 

s 



C5 
Z 



CM 
CM 



O 2 
O 2 



The INPUTS described in 
the "given" portion of a 
statement of objectives 
involve {combinations 
of): 

-Relatively little 
content 

-Relatively easy learnirg 
problems 

-Recall 


AND 

The ACTIONS described in 
the "student wl 1 1" 
portion of a statement 
of objectives involve 
(combinations of): 

-Relatively short chains 

-Relatively easy 
learning oroblems 

-Recall 


^ CO 


The INPUTS described In 
the "given" portion of a 
statement of objectives 
1 nvol ve (comb 1 nat 1 ons 
of): 

-Relatively much content 

-Relatively difficult 
learning problems 

1 -Transfer 


AND 

The ACTIONS described In 
the "student will" 
portion of a statement 
of objectives Involve 
(combinations of) : 

-Relatively short chains 

-Relatively easy 
learning problems 

-Recal 1 


g 

43 

CO 
CO 

1 

CO 

a 
1 


all nemberB of each 
INPUT clasB 

'Plan to require all 
membert of ACTION clasa 


The INPUTS described In 
the "given" portion of a 
statement of objectives 
Involve (combinat Ions 
of): 

-Relatively little 
content 

-Relatively easy learning 
problems 

-Recall 


AND 

The ACTIONS described In 
the "student wi 11" 
portion of a statement of 
objectives involve 
(combinations of) : 

-Relatively long chains 

-Relatively difficult 
learning problems 

-Transfer 


CO 

CO 0 ^ 
0 • «o 0 

t. ttt 

^ to 

ft, t 

» L 


4J» 

0 

ID CO 

V to 

E t) 


The INPUTS described In 
the "given" portion of a 
statement of objectives 
Involve (combinat ions 
of): 

-Relatively much content 

-Relatively difficult 
learning problems 

-Transfer 


AND 

The ACTIONS described In 
the "student wi IT' 
portion of a statement of 
objectives Involve 
(combinations of): 

-Relatively long chains 

-Relatively difficult 
learning problems 

-Transfer 


1 

CO 

V) 

1 

CO 

ft. 


t. 

KS to 

0 4* 

' to CO ^ 

§ § 

E ^ 

•-AS* 
0 H fx, 


>: 
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^ CO 

■f § 


CONDITIONS 


ACTION 
TO TAKE 
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F.2.2 



ILLUSTRATIOM SUMMARIZING PROClDURES FOR DETERMINING 
THE SIZE OF THE SAMPLE OF CRITERION BEHAVIOR 
TO DE REPRESENTED IN THE CRITERION TEST 



#1 



FOR EACH CRITERION BEHAVIOR: 

a. Inspect FORM D.2{l ) 
(statement of objectives) 
for: 

a. I Transfcr/rec - 1 J 
requ i rcments 

a. 2 Number of classes 

of i nputs ident i f ied 

a. 3 Number of members 

per class fdentiffcd 

b. Inspect the collection of 
task analysis forms 
(A.5(^) or (11)] for: 

-The length of chain 
i nvol ved 



n 



Decide on the number 
of classes of INPUTS 
to be represented 

Decide on the numbt — 
of test items per 
class to be developed 



FORM A.5('#) or A. 5(11) 



FORM D.2(l) 



0 




□ 



\ 



iL 



J 



— c^-l 1 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


The criterion test containing one or more items vith 
epecij ccations for cm item spelling out: (2) vnat the 
student will be given; 12) what he will he expected to 
do; and (^) what he will be expected to produce. 


WHAT YOU WILL 
WORK FROM 


(1) Statement of objectives re: each criterion 
behavior 

(2) Simulation decisions re: the objective 

(3) decision about how much behavior to sample and 
how many items to develop. 


WHAT YOU WILL 
DO 


(1) Prepare criterion test Item(s), 


FORMS YOU WILL 
USE 


FORM F.2(l) for developing test Items. 
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DESCRIPTION OF Sub STEP 



F.2.3 



INPUT 



ACTION 



OUTPUT 







1 rcpnro criLcrioii verity 






criterion objective 




iten(s) 




containing : 


saapling decision 








•GIVEN 










-STlJDraT WILL 


sinulation decisicn 








-RESULTING IN 








vii 




viii 




ix 



Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 



-MATRIX: Types of 


-MATRIX: Making 


-MATRIX: 


Adequacy 


FORM F.2(l) . 


. 65 


test 5 terns . * 51*57 


scorinq 


of test 




SUMMARY OF 
PROCEDURES . 




"objective** . 59-61 
-m;>,tr|X: What to 
Include in items 
and where to get 
Information • . . 67 


i terns 


.... 69 


. 66 



RequirofJ Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Samp 1 i ng dec! s i on 


F.2.2 


Col lection of FORMS 
carried forward from 


F.2.2 


FORM F.2(1) 


Selection of 
simulat ion plan 
(where dppl icab le) 


E.2.3 
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Sub-ST£P i r.2.3 1 
















ACTION 




OUTPUT 




Statenent of 
criterion objective 

sampling decision 

simulation decision 

vii 




Prepare criterion 
test iteTn(s) 




CRITERION TEST item(s) 
containinn^ 

-GIVEN 






viii 


► 


--STUDENT WILL 
•RESULTING IS 

















CRITERION 

STATEMENT OF OBJECTIVES TEST ITEM ELEMENTS 



GIVEN 



Translate into 
test i tern content 



vi f .a 



vi n.a 



INFORMATION TO BE 
GIVEN THE STUDENT 



Ix.a 



STUDENT WILL 



vM.b 



Tr<ins late Into 
test i tern content 



V I i i . b 



WHAT THE STUDENT IS 
EXPECTED TO 00 



ix.b 



RESULTING IN 



vi i • c 



I 
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Translate Into 
test i tern content 



V I i f . c 



50 



-WHAT THE STUDENT IS 
EXPECTED TO TURN OUT 

-STANDARDS 



Ix.c 



TYPES OF TEST ITEMS* 





page 


How a test item samples 
criterion behavior 


52 


What a test item tests for 


53 


Test i terns rcqui ring 
rccal 1 vs . transfer 


5^ 


Three elements to a test item 


55 


Three possible modes of response 
In test items 


56 


Desirable properties of test items 


57 



^Thc phrase "test item^' is meant to include amj measurement 
eituation (whether a paper-and^pencil test, an assignment 
involving objects an^l/ov other people, a procedure) 
designed to assess acquisition or retention of criterion 
behavior. 
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F.2.3 



criteria for identifying how a test iteh 
sahples criterion pehavicr 



lOENTtFICATlON 
MATRIX 



DIFFERENCE 
BETWEEN 
"TOTAL" 
:R ITER! ON BEHAVIOR 
AND A 
TEST ITEM 



CRITERIA 



GENERAL CASE 



"TOiAL" CRfTtRION BEHAVIOR 
represented in a 
task analysis diagram 



-A task analysis diarjrccx represent 
all tlv clopnfts (both of INPUTS 
m3 of ACTIONS, which the 
perfomcr mcaj encounter on a test 
or on tkc Job 

As such it (iocs not fleet the 
real vor;! in uhich only one 
.nenbcr of a class is usually 
encountered 



INPUT 



ACTION 



OUTPUT 



x-2 _J 



-co- 



rn 



ED- 



A TEST ITEM 
testinq criterion behavior 



test item presents (usually) 
only onr. menber of a class 
(INPUTS or ACTIONS) at a time 



-As such it reflects the real 
t>orld in which criterion behavior 
requires dealing (usually) with 
one instancr* at a time 



GIVEN: f2 
STUDENT WILL 



Produce either 
u • I or u • 2 



EXAMPLE 
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INPUT 



I you | - 



she 



ACTION 



52 



GIVEN: "he" 

STUDENT WILL: Use "Is' 



F.2.3 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING WHAT A TEST ITEM TESTS FOR 



1 



WHAT IS 
TESTtO FOR 



CRITERIA 



Capabi 1 i ty 
of exhibi ting: 

the shortest possible 
cha In , 
I.e., a s imple 
associ at ion 



Capab 1 1 1 ty 
of exhibiting: 

a chain of 
Intermediate length, 
i .#», , a series of 
associations 



CapaM 1 1 ty 
of exhibi ting: 

a very long chain. 
I.e., a long series 
of associations 



'The student ia given 
one criterion INPUT 
in a test item 

-To be able to exhibit 
the chain, he must 

Oi 3 criminate the 
input from other 
possible inputs 

**When the input 
belongs to a class 
of inputs, he must 
generalize to the 
test input (if not 
previous ly 
encountered) 

• * Produce the action 

associated with the 
input (and not any 
other) 

or 

• * Produce an 
alternative action 
(generalize) if 
another ia allowable 



The intermediate case 



"The student is {riven 
one criterion INPUT 
in a test item 

"To be able to exhibit 
the chain ^ he must 

••Do everything in 
Column 1 

••Do everything in 
Columri 2 repeatedly 
as each action 
px'oduces an output 
which beetles the 
input for another 
action 

••Do all the 
repetitions in the 
correct sequence 



EXAMPLES 



O - 
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GIVEN: 



The INPUT 
"the boy" 



STUDENT WILL: Produce 
the ACTION 
(saying) - 
"walks" 



GIVEN: The INPUT - 
a quadratic 
equation and 
Instructions 
to factor It 

STUDENT WILL: Take 

the series of 
act Ions 
involved in 
factoring 



GIVEN: The INPUT - 
a specific 
novel to read 
and i ns tr uc- 
tfons to wr i te 
an essay about 
it 



STUDENT WILL: 
series 



the 



Take 
of 
ACTIONS 

requ I red to do 
this 



CRITEKIA FOR IDENTIFYrNG TEST ITEMS OR TEST PROCEDURES 
INVOLVING RECALL VS. TRANSFER 



IDENTIFICATION 
MATRIX 



TYPE OF 
REQUIREMENT 


RECALL 
of 

INPUT AnH/or AtJtCiti 


TRANSFER 
for 

iNrui ana/or ACTION 


CRiTERiA 


The "GIVEN" presents a student 
with an INPUT already encountered 
during instruction, i.e., an old 
input 

AND/OF 

The "STUDENT WILL" requires the 
student to recall an ACTION 
already practiced during ins true- 
tion, i.e., an old action 


The "GIVEN" presents a student 
vith an INPUT not previously 
encounterrt*^ firing instrur*tion, 
i.e. , a new input 

AND/OF 

The "STUDENT WILL" requires the 
student to transfer to an ACTION 
not previously practiced during 
instruction, i.e., a new action 









ENGLISH - 


1 

Noun/verb agreement 




e.q. » u)fiert fie goz^ 


to vooKk, 




e.p., when ^za^chinq f^oK f^ood. 




man 










EXAMPLES 


take a 








kutiti at nic(ht 




take^ a 


ba6 






ha.)it at Yiij^ht 




Both the INPUT 


'man 


- and the 


ACTION 


Botii the INPUT -tLqt^" and the 




associ ated wl th 


i t 


--tafecA- were 


ACTION associated with \t -fiuntA- 




used during ins 


true 


tion; th» 


re fore, 


NOT used during instruction; 




both Input RECALL and action 


RECALL 


therefore, both input TRANSFER and 




are requi red 








action TRANSFER are required 
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F.2.3 

I0ENT9FICATION 
MATRIX 



CRITERIA FOR IDENTIFYING THREE CONTENT ELEMENTS FOR A TEST ITEM 

OR TEST PROCEDURE: (THEY PARALLEL THE THREE ELEMENTS 
IN A STATEMENT OF OBJECTIVES): 



TEST 
ELEMENTS 


"GIVEN" 


"STUDENT WILL" 


"RESULTING IN" 




Jnjoi^cition you plan 
to give to atudente: 


wnat students are 
expected to do: 


What students are 
expected to turn out: 




~In8tT*uction3 about 
what to do 


-Actvon(s) to be taken 
(new or old) 


-Outputs to be 
produced 


CRITERIA 


^Question or problem 

••Criterion input (a) 
to hp, presented 
(new or old) 

• •Aids to he 
presented (if any) 


• •Mode of action 
(recognition, 
editing, production) 


• •Answers 

• • Products ( tangible , 

non-'tanaihle ) 

^Type of scoring 

^Standards for scoring 
above 




AND 






-If recognition test 
is to be used: 








••Answer options 
(i.e., actions) 







o 
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F.2.3 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING TEST ITEMS OR TEST PROCEDURES 
INVOLVING THREE RESPONSE MODES 



TYPES OF 
TEST ITEMS 


RECOGNITION 


EDITING 


PRODUCTION 




The "GIVEN" cant cdns: 


The "GIVEN" contains: 


The "GIVEN" contains: 




--^ 8 tate^ent oj a 


"A sta temen t oj a 


"'A statement of a 




question or problem 
which includes: 


question or problem 
which includes: 


question or problem 
which includes: 


CRITERIA 


• •Cri tenon INPUT (S) 
AND 

--Answer options (the 
criterion action 
associated with the 
input plus false 
choices) 


• • Cri t^nnn INPUT (^ ) 
AND 

^A(^ION associated 
with it: 

'•Wrong action (to be 
corrected) 

••Correct action (to 
be left unchanged) 


••Criterion INPUT (S) 




The -'STUDENT WILL" 


The ''STUDENT WILL'' 


The ''STUDENT WILL" 




requires th^ 
student to: 


requires the 
student to: 


requires the 
student to: 




^Select from among 


^Edit or correct an 


-Produce the ACTION 




the option^* * 


incorrect action 


associated with the 




• •Checking correct 
option (i.e., ACTION 


^Leave a correct 


INPUT 




action alone 






associated with the 








INPUT) 







EXAMPLES 


e.g. , yohick of^ tkz 

do Lfou oAc vohtn 
thz 6ubjzct o(i a 
6znttnct -C6 "it"? 

am oAe 




e.p. , mafee up a 

sentence uAinQ 
the, iOon.d6 ''it" 
and thz pn^oMznt 
te,n6z Oj{ thz 
vcAb "to be" 
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IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING OES!RABLE PROPERTIES OF TEST ITEHS 



PROPERTIES 



CRITERIA 



CONTENT 
of test items 



'Should reflect the difficulties 
involved in learning and perform^ 
ing the criterion behavior; i.e., 
t/>pt items ^^hould scFnvlc difficult 
or error^prone: 

• * Discriminations 

• * Generalizations 
• 'Associations 

Chains 

-Recognition and editing items 
should: 

Concentrate on error^prone 
behaviors 

Concentrate on behaviors 
involving learning difficulties 



FORM 
of test items 



'Written test items should: 
**Be concise^ not wordy 
**Be clear, unambiguous 
• •Not be tricky 

••Pro'Hde adequate instructions 



'Recognition items should: 

**Have problem completely stated 
in the stem (not in the options ^ 

••Provide plausible options 

• •Preferably provide four options 
( to minimize chance factors in 
netting answer) 
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TYPES OF SCORING PROCEDURES 





page 


Conditions making for 
objectivity in scoring 


60 


What to do to make 
scoring more objective 


61 
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lOENTIFICATJON 
MATRIX 



CRITEKIA FOR IDENTIFYING CONOITIONS THAT MAKE 
SCORING OF TEST ITEMS SUBJECTIVE OR OBJECTIVE 



SCORING 


SUBJECTIVE 
when 


OBJECTIVE 
when 




Evaluation of outp-uts 
ia norc difficult bccaucc: 


Evaluation of outputs 
is easier because: 




-Conparison with etcmriard crutput 
poses discriv^iruztion problcns: 


-Comparison with stcyvmrd outvut 
poses lesser discriniiiation 
problems: 




**No model output is available 


••A nodel output is^ available 


CRITERIA 


• ^Properties of standard arc not 
ideyitified 


Properties of standard are 
identified 




**The student-produced output is 
transient 


* •The student-produced output is 
perstiunent 




-Evaluation tack creates a more 
co^ Heated discrimination 
problem because it requires : 


-Evaluation task creates a less 
complicated discrimination 
problem because it requires*' 




• •Judgment about degree 


• •Judgment about simple presence 
or idjisence 




• •Estimating or rating 


• • Counting 




• 'Measurement of properties 
unaided by measuring instruments 


• 'Measurement of properties 

aidsd hu m^cf^i/rin,^ in^trtsv^nts 




• 'Measurement of psychological 
properties 


••Measurement of physical 
properties 








e.q. , an t66ajf 


t.g*, asumx to a matli p\oblm 


EXAMPLES 


c..g.p an intcApX(Lt(V(>ion 0|( an 
-c44ue <n lvc6toxu 

<L*g., the. quatUu movojn^nt^ 
in ^po^uU 


e.g., punctudtion of^ a 4e»ifence 
w^M^h^ a pK^CAJb^ momnt 
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F.2.3 

DECISION 
MATRIX 



DETERMINING HOW TO MAKE SCORING MORE OBJECTIVE 



CONDITIONS 



ACTION 
TO TAKE 



Discriminations involved in 
evaluation are likely to be nx3rc 
difficult because of subjectivity 
concern! nq standards 



-Produce a standard for cariparison 
iMcnever possihlcy and/or 

'Identify the properties of a 
standard output: 

••/l list of repuired properties 

'Create a pcmancnt^ tanaible 
record of the student output^ 
'if possible; ^.a., 

••/I film of motor hehai'ior 

••/In audio tape of speech 



Discriminations i nvol ved i n 
evaluation aic likely to be more 
difficult Lecause of subjectivity 
in the evaluation task 



-Strive to evaluate on the basis 
of simple presence or absence 
(when possible) 

-Provide bench marks for degrees 
when it is necessajvf to: 

••Rate 

• • Estimate 

'Identifu an objective ^ behavioral 
output that represents the 
psycholo^iical property being 
evaluated 
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JOB PROCEDURES 





page 


Form to use in preparing 
test items 


65, 66 


Information to review for 
preparation of test items 


67 


Summary of procedures 


68 


Stand2»rds for test 5 terns 


69 



Er|c ^^^63 



Form F 2111 



LiSSON 



0BJ€CTIV£ 





INrOrlAiATIOPi YOU PLAN TO UlVc TO STuDcNT 


» Instructions 




— Ou«t»on or 
Froblonf^ 




• • 1 §^Pl J TC 

(nM^^old) 




••AIDS 




• •ANSWER 
OPTtONS 
(«^«n applicable) 





••STUDENT WILL" 



WHAT THE STUDENT IS EXPECTED TO DO 



- ACTIONS 
• •mode 



'HESULTINQ IN^ 



¥WAT THE STUDENT IS EXPECTED TO TURN OUT 



- OUTPUT 

• • antiMtrs 

• •proctuct 

* Typa ol Scoring 



LiSSON 



OBjicrive 



IMFOBMATION YOU PUM TO GIVE TO STUOCNT 



^Qvmt»of^ or 
••INPUTS 
••AIDS 



OPTJOWS 



•*STUO£NTW1LL' 



WHAT THE STUDENT IS EXPECTED TO DO 



- ACTIONS 



"RESULTING IN'' 



WHAT THE STUDENT IS EXKCTED TO TURN OUT 



• OUTPUT 



• •proilMCt 
«»Ty0fto^Sc(K(nf 



I C *^**^ 



46 



F.2.3 

DECISION 
MATf^lX 



OETERHINIM WHAT INFORMATION TO REVIEW 
IN PREPARING CRITERION TEST ITEHS 



INFORMATION 
NEEDED 


Wh«t to include in « 
criterion test item: 

••GIVEN 

••STUDENT Will 
••RESULTING IN 


Hoi* much of criterion 
behevior to sample end 
with how many 1 tem» 


WhetKer to test for 
criterion L^hevior or 
for Sfmul«tio<) uT 
cr 1 terion behevior 
• 




Rcvie:^: 


Reviev : 


RevimJ: 


ACTION 
TO TAKE 


^Statement of 
criterion objective 
on FORM D.2(l) 

if neceeaary, 

-AsBociated task 
analysiB reeulta on 
collection of FORMS 
A. 5(4) or (11) 


'3taterfmnt of 
criterion objective 
on FORM D.2(l) 

and 

'Decision nade in 
Suh^STEP 


Simulation decision 
made on SubSTEP E.2.3 
(baaed on alternate 
plans recorded on 
FORM F.Kl) 
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STANDARDS 
MATRIX 



CRITERIA FOR ASSESSING THE ADEQUACY OF CRITERION TEST ITEMS 



PROPERTIES j 


CONTENT 
COMPLETENfSS 


REFLECTS 
PRIOR DEC 1 SIONS 


FORMAL 
PROPERTI ES 




'Three clenents 
prcparcil 


^Statenent of 
criterion ohjecrtives 


'Lenlc itself to 
objective ecorifVJ 


CRITERIA 


• • CTVFN 


-Simulation decision 


'Brief, concise 




m m C^^J IP L'fJT LffTT 
• '^-t UL--^.iI 9^1 LI J 




•0 Leczr , wic2jtI[?Z(Juous 




"RESULT nr. J,V 








'Content fcr three 
elcmcntr, aarplrtc 
[scc FOm F.2(l) ] 


1 





ERIC 



69 



PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A teat containina items teetina for subcritevion 
objectivee. 


WHAT YOU WILL 
WORK FROM 


(1) Statements of subcrfterlon objectives. 

(2) Decision about how much behavior to sample and 
how many items to develop* 

(3) Decision whether or not to simulate. 


WHAT YOU WILL 
DO 


(1) Develop subcrfterlon test items. 


FORMS YOU WILL 
USE 


FORM F,2{1) for developing test items. 
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INPUT 



subcriterion 
objective 

sanplin/:; decision 

simulation decision 



DESCRIPTION OF Sub-STEP 



F.2.4 



ACTION 



Prepare subcriterion 
test item(G) 



XI 



OUTPUT 



SUBCRITEHION 
test item(s) containing: 

«STUD}^]JT WILL 
-R>:SULTING IN 

xii 



CRITERIA FOR 
IDENTIFYING INPUTS 



Job Aid ('ontenls 
ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: 
01 f ference be tween 
criterion and 
subcriterion test 






FORM F-2(l) 

SUMMARY OF 
PROCEDURES . . 75 



Requinui Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Sampl tng dec is Ion 


F,2.2 


Col lection of FORMS 
carried forward from 


F.2.2 


FORM F.2(l) 


Selection of 
simulation plan 
(where annl (cable) 
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Sub- STEP 



F.2.4 



INPUT 



Statenent of 
subcriterion 
objective 

samplin/^ decision 
+ 

simulation decision 



JOB DIAGRAM 



ACTIOM 



Prepare subcriterion 
test itein(s) 



XI 



OUTPUT 



SUliCRITERIOn 
test ite:7i(s) containing: 

-GIVEN 

-SrJDENT WILL 
-RESULTING IN 

xii 



STATEMENT OF OBJECTIVES 



GIVEN 



x.a 



Trans late I nto 
test ] tem content 



XI . a 



SUBCRITERION 
TEST ITEM ELEMENTS 



INFORMATION TO BE 
GIVEN THE STUDENT 



xi I .a 



STUDENT WILL 



x. b 



Translate into 
test i tern content 



xi .b 



WHAT THE STUDENT 
EXPECTED TO DO 



IS 



xi i .b 



RESULTING IN 



x. c 



Trans late into 
test i tern content 



XI .c 



-WHAT THE STUDENT IS 
EXPECTED TO TURN OUT 

•STANDARDS 



xi i .0 
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JOB PROCEDURES 





page 


Difference between criterion 
and subcriterion test items 


7^ 


SUMMARY OF PROCEDURES 


75 



















o 
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F.2.4 

IDENTIFICATION 
^AATRIX 



CRITERIA FOR Dl STI fiGU I SH I NG BETWEEN 
CRITERION TEST ITEMS AND (UNHODIFIEO) PREPARATORY TEST ITEMS 



TYPE OF 
TEST ITEM 


CRITERION 
test item 


SUDCRITERION 
test Item 


CRITERIA 


The teat item tacts for tkc 
entire chain iyivolvcd in 
criterion behavior* 


The test item teats for onlij part 
of the chain; but it ic an intact 
part^ i.e., tlie criterion behavior 
is not modified (except for scope) 
in arty way 



^iJOTE: Each item tests for only one of the many possible voi^'iations a criterion 
cfiaui ca)i have; hut it still tests an entire chain. 



IMPLICATIONS 



Considerations in developing test items for subcrlterion 
behaviors, except for scope, are Identical with those involved in 
preparing criterion test ( temn; (See Sub-STEP F.2.3) 

I 
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ILLUSTRATION SU^^MA^xl2 ING PROCEDURES INVOLVED IN PREPARING 

SUaCRITERION TEST I TEHS ^REGAROI NG UNMODIFIED PARTS 
OF CRITERION BEHAVIOR) 



/I 


n 




REVIEW 




DEVELOP SUBCRITERION TEST ITEMS 






a. Statement of subcrlterion 
objectives on FORM D.2(l) 
for three elements of a 
test item 

D« uecision aoout sampling 
of subcrlterion behavior 
(Sub-STEP F.2.2) 




a. Spell out the '^GIVEN" 

b. Describe the "STUDENT WILL" 

c. Provide the "RESULTING IN" 
i nc luding standards 






c. Decision about whether 
to use simulation (in 
Instruction and in 
testing) of subcriterion 
behavior (and how to 
simulate) on Form E.l(l) 













FORM D.2(I) 



FORM E.l(l) 



FORM F.2(l) 



I I I r- - 




LLQJJ □ o □ 



© 
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STEP 



COMPLK I ION CHECKLIST 







IDENTIFIED 


PERFORMED 


PRODUCED 


FORMS COMPLETED 


F.2.1 




Dec i s . jn made 
whether to assess 
cr i teri on and/or 
subcri terlon 
outputs 









Decision made about 

how much of 

cri terion behavior 

to sample and how 

many i terns to 

prepare 











Cri terion test i terns 


rORM F.2{I) 








Subcri teri on test 
i tefns 


FORM F,2(I) 
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STEP 



F.3 



F,3 



Develop diagnostic tests to identifj^ reason for failure to acquire the 
criterion behavior taught in each instructional unit. 



Plan the use of results on criterion test iteas and on 
subcriterion test itens in diagnosing type of learning failure that 
has occurred. 



F.3.2 



Develop diagnostic interview procedures to provide supplementary 
diagnostic infomation about results on criterion test items and 
on subcriterion test items 



F.3,3 



Develop diagnostic test items designed to identify specific types 
of learning failures. 





STEP 



F.3 



Statement of objectives 
and associated analysis 

criterion and 
subcriterion test items 
for each criterion 
behavior 



OVERVIEW 



ACTION 



Plan the way to 
diagnose learning 
failures regarding 
the criterion behavior 



OUTPUT 



Plan Involves: 

-Use of criterion tests] 

-Use of subcriterion 
tests 

-Development of 
diagnostic items 



Subcriterion 
F.3.1 objectives associated 
with a criterion 
behavior 

■f 

developed criterion 
tests and preparatory 
tests 



Review to determine 
adequacy of criterion 

and subcriterion tests 
to serve a dialectic 
purpose 

ii 



DECISION 

-Criterion tests and 
subcriterion tests 
can diagnose problems 
AND/OR 

-Diagnostic tests aff^ 



statement of objectives 

F.3.2 * 

task analysis results 

learning analysis 
results 

criterion test items iv 



Develop diagnostic 
probes vhen necessary 



PROBES 

To diagnose type of 
learning failure and 
sources of the failure 



Tl 



Statement of objectives 

F.3.3 task analysis results 

learning analysis 
results 

criterion test iteas 
vjjl 



Develop diagn-ostic 
test items appropriate 
to anticipated types 
of learning failures 



viii 



DIAGNOSTIC TEST ITEM 
to diagnose tjpe of 
learning failure 



Ix 



STEf 



PAGE INDEX 



CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 



F.3.3 



-MATRIX: 
Non*di agnostic 
nature of 
single item 
results . . . 86-88 


Multiple ttttt 
ifm to di«9nott 

. . 31-103 

-MATHU. 
Inform*! io.i 
to rcwit«r in 




SUMM,.RY OF 
PROCEDURES . . .107 




-MATRIX: Using 
probes regarding 
criterion tests 
vs« diagnostic 
tests Ill 


-MATRIX: Types of 
probes to 

use . • . 116-121 
-MATRIX: Oevelopinq 
probes . . J23-I23 


-MATRIX: Adequacy 
of probes which 
have been 

developed . . . 1 7,9 


SUMMARY OF 
PROCEDURES . . 128 




-MATRIX: Types 
of diagnostic 
test 

Items . . . 133-137 


^rioritiet 
•r4cr of 

M4mini tt»r i ■ i a 
ictt itentt ... |X«0 
*ilATfc 1 K ; 
lnform«t ion 
to roviow aft 
• fcotii for 
4*vtlopif)^ 

4i ftf r>e>^i U III 
l«it ... IHi 


-HATRIX: Adequacy 
of diagnostic 
test items . . Ilt3 


F.2(l) 

SUMMARY OF 
PROCEDURES . . \k2 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


Deter r^nina whether tcets arc ccrpahle of diaanL.3ing 
the ti/pea of learning failures which have occurred. 


WHAT YOU WILL 
WORK f ROM 


(}) Subcriterion and criterion objectives. 
(2) Subc<^I teflon and criterion tests. 


WHAT YOU WILL 
DO 


(1) Revlov test items lo deterniinc whether anticipated 
learning failures can be diaqnosed by them. 

(2) Decide vyhether the existing tests are adequate 
or inadequate for diagnostic purposes. 


FORMS YOU WILL 
USE 


No'^e 
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DESCRIPTION OF Sub STEP 



F.3.1 



I 



Subcriterion 
objectives associated 
with a criterion 
behavior 

developed criterion 
tests and subcriterion 
tests 



AC't iON 

Review to determine 
adequacy of criterion 

and subcriterion tests 
to serve a dia^^iostic 
purpose 

ii 



OUTPUl 

DECISION 

-Criterion tests and 
subcriterion tests 
can diafrnose problems 

And/or 

-Diagnostic tests au-e 
needed ^ ^ 



Job Aid Contents 



CRITERIA FOR 
IDENTIFYING INPUTS 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: 
Non-diagnostic 
nature of 
single item 
results . . . 86-88 



-MATRIX: Using 
multiple test 
items to diagnose 
learning 

failures . . 91-103 
-MATRIX: 
Informat ion to 
review in 
planning 

diagnosis ... 106 



SUMMARY OF 
PROCEDURES 



107 



Required Malonials 



COMPtCTCD MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


Bl ANK FORMS 


Analysis materials 
and statement of 
object ives (carried 
forward) 


F.2 


Crl terion and 
subcriterion tests 


F.2. 4 


































C — 
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Sub- STEP I F^a^n 



JOB DIAGRAM 



INPUT 



ACTJON 



Subcrlterion 

objectives associated 
with a criterion 
behavior 

♦ 

developed criterion 
tests and subcritcrion 



tests 



i 



Review to detemine 
adequacy of criterion 

and subcritcrion tests 

to servo a d^a:':iiostic 
puri^ ->se 

ii 



Objectives identi fy 
potential learning 
problems 

♦ 

criterion and 

subcritcrion tests 
Mkcly to be ADEQUATE 
to diagnose them 



i.a 



Do nothing 






ii.a 



Objectives Identify 
potential learning 
problems 

criterion and 
subcritcrion tests 
likely to be INADEQUATE 
to diagnose them 



Prepare special 
diagnostic i terns 



n.b 



OUTPUT 



DECISION 

•Criterion tests and 

subcritcrion tests 
can d\»i^nor>e problems 

AJ^D/OR 
►Dia/^nostic tests are 
needed 

ill 



Cri terlon and 
subcritcrion tests 
to be used for 
DIAGNOSTIC purposes 



ill. a 



DIACNOSTIC test Items 
to supplement the 
cri terlon and 
subcritcrion I terns 



INABILITY OF USING RESULTS 
SINGLE CRITERIOIJ OR SUBCRITERION TEST ITEMS 
TO DIAGNOSE LEARNING FAILURES 





page 


Type of diagnostic information 
you wish to obtain 


86 


How the same wrong ansv,*cr 
i.^ay be due to a variety of 
learning fai lures 


89 
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F.3.1 

IDENTIFiCATION 
MATRIX 



CRITERIA FOR IDENTIFYING TYPES OF DIAGNOSTIC INFORMATION YOU WISH 
TO OBTAIN: WHAT IS THE SOURCE OF FAILURE 



SOURCES 
OF FAfLUHE 



CRITERIA 



Re: INPUTS 
'Discriminat ions 
'General izations 



Has the c indent 

'Treated differ ent classes of 
inputs in the sane vay when he 
should have: 

• • Discrimuiated anong the classes 
AND 

•'Taken different actions in 
response to them 

'Treated menhevs belonging to the 
sane class of inputs in d iff eren t 
ways when he should have: 

••Generalized across the members 
AND 

••Taken the sane action in 
response to them 



Re: ACT'ONS 
•General i zat ions 
-Associat ions 
-Cha ins 



lias the student ^ 

^Failed at action generalization: 

••Failed to exhibit an 
alternative way to take an 
action 

••Taken an incorrect alternative 
action 

^Confused which action goes with 
which class of inputs when he 
should have: 

••Properly associated the right 
input with right action 

-Failed to complete a chain or 
failed to perform its parts 
in the correct seaucnce 



EXAMPLES 



See opposite page 
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F.3.1 



EXAMPLES OF LEARNING FAILURES 



EXAMPLES 


Re: INPUTS 


Re: ACTfONS 




A DISCRIiMllJATION fai lure 


An A5G0CIATS0N failure 




e.g., 6tad(int mA^tc^ 'T/ic a'^ncn 
lb 6ldz'' 


e.g., ^^tudcnt iO^XA^tc^ 'T/ic ioomcn 


lUGLlSU 


-Failure to discriminate between 


-Failure to associate 


noun/vQAb 


''mmoji'' and "looirjcn" 

belonging to the two different 
classes of inputs : 


ptun^at nounb and the. pZivxal 

p^O^W 0^ tliC, \JCAb 




-Has treated the two classes as 
if they were the same 


-Has associated the wrong action 
with the Input presented to him 




A GENERALIZATION fai lure 






C-fl., student wiltcb 'T/ic ii)om(in 






-Fails to generalize across 






KdriulaA and t^iKCQuZoA plU'^Rtb , 
att 0^ idUdi bdtonq to tiia 
^>CUrtZ c^at>6 






-Has treated members of the same 
class as if they were different 




NOTE • 


The same error, a^.Z., W^AXina can be indicative of thre»« 
different types of learning failure. By itself, the result is not 
diagnostic. It is indetermanate which typ^e of error has been made. 
See discussfon on next p^'^ge. 



o 
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F.3.1 

SOENTIFICATIUISI 
MATRIX 



CRITERIA FOR IDENTIFYING THE NCN-D I AGNOST I C NATURE 
OF RESULTS ON SINGLE CRITERION TEST ITEMS 



TYPES OF 
FAILURE 



CRITERIA 



DISCRIMINATION 
among i npots 



GENERALIZATION 
across inputs 



ASSOCIATION 
between input cind acticn| 



On a criterion test itcn all the student is required to do is to 
produce a cr*itericn ACTIOW Any of the three types of faihire will 
shou up as a uro? r; (or omitted) action. The sa\ne wrong action can he 
due to Ofiy of the th^^c failures. Therefore j its source ''s 
indeterminate. | j 




A student is given the following test item: 

A 6t/ie>S6 ha6 been (^^jn pldct o.^ papcA {iX >66 noto 

cAimbtcd] . I<s tJiAj> a pznf^dcXJCij clas tic o^ a nor- pcxitcJUiu clastic 
object? 

The student gives the wrong answer; takes the wrong ACTION: 

S01J6 It AJi "poA^cctlij clastic. " 

The failure may be due to an Inability: 
-To discriminate between examples of 

pzA^ccXiM clattic and non-poAizctZif cJUJU!>tJiC objtctJ^ 

OR 

-To generalize across varying (dissimilar) examples of 
non-pch^lcctZif ^Znhtic objtctA 

OR 

-To associate the label 

*'non-p2A^ccttij eZ<utlrJ' i^itli an cxamptc 



*See previous page for another example. 
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HOW TO USE MULTIPLE 
CRITERION OR PREPARATORY TEST ITEMS 
FOR DIAGNOSTIC PURPOSES 



Problems re: 

INPUTS INPUTS fm ACTIONS 

••Discriminations • • Assoc i at ions 
• •General i zat ions 


page 


How multiple test items sample inore 
varieties of the criterion behavior 


91 


Properties of test items and results 
which can be used diagnostical ly 


92 


Interpreting consistency vs. inconsistency 
i n error patterns 


9'*, 95 


Interpreting some general patterns of 
resul ts* 


96 



^The nmher of comhinations and pcTtnntationG (of: right 
and wrong answers, types of wrong wwvcrs, and kinds of 
itaris) is too Im^gc to be able to idc^ntify them all and 
to interpret than iineqidvoeallij. Thin section and related 
sections in Part J on "the evaluation of instructional 
materials y" therefore, only identify patteims of results 
which lend themselves to interpretations which can he made 
with reasonable certainty. 
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F.3. 1 



CRITERIA FOR IHENTIFYiNG WHAT TYPE OF "MULTIPLES" 
ADDITIONAL TEST ITEMS SAMPLE 



IDENTIFICATION 
MATRIX 




NUMBER OF 
TEST ITEMS 


A SINGLE 
test item 


MULTIPLE 
test items 



CHITEKIA 



^Whcn there arc nultipln clar.scs 
of INPUTS or of ACTIOUS, and/or 

^Vkcn there are nultiplc nernhero 
Mthin each clacc, 

A siywle test iten usually A multiple tect item uaually 

sanples: sarrpleG: 

••Only one claos, or ••More than one clacs, or 

••Only one met^.her within a elans ••More than one rnemher of a class 



GENERAL EXAMPLE 



INPUT 



ACTION 



OUTPUT 



A single test item would normally 
present only one of the five 
diagrammed INPUTS 



Multiple test items might present 
at least one input for each of the 
two diagrammed classes of inputs, 
and possibly two inputs per class 



EXAMPLE 



HE 



SHE 



IT 



A single test item might present 
the student with the pronoun "sha' 
(and require a sentence with "is" 
in It) 







m 1 





YOU 




ARE 





IS 



Multiple test items are likely to 
present all five pronouns as INPUTS 
(with the student required to 
produce the associated ACTION, i.e., 
usinp the right verb) 



F.3.1 


CRITERIA FOR IDENTIFy'NG PROPERTIES UF CRITERION 
(OR SUBCRJTERION) TESTS WHICH HAVE DIAGNOSTIC CAPABILITIES 


IDENTIFICATION 
MATRIX 






TYPES OF 


Independent variables 


Dependent var iablcs 


r KUr LK M 




meosurcd by you 




of itms presented to 


^Nature of errors made by 
students 






••Takinp the, same action for 
inputs requiring different 
actions 


CRITERIA 


-Number of items presented to 

sUiJiicnti? 

••Number of classes (inputs/ 

liCUi^Uni^/ CdCrl I^eprC SCrtZ-CCl Dy CL 

test iten 

••Number of members per class 
each represented by a test item 


••Taking different actions for 
inputs requiring tlie same 
action 

• •Mismatching inp^it and action 
••Actions are omitted altogether 

-Consisteiicy of errors made 




••Nwnber of test items per class 
(input/action) 

••Number of test items per each 
member of a class 


•^rhe sane/different type of 
error made on recall and on 
transfer^ items which are 
parallel to one ancther, <^-{7*> 
represent the same INPUT class 




-Reversal in the direction of 
pci^fomance required by test items 

••Criterion INPUT is the '*given** 
in the test itemj criteHon 
ACTION is what the ''student 
will" do 


••The same/different tjpe of 
error made on more than one 
xtcm per class or per member 
of a class 

••The same/different error made 
in a reversal item (if used) 




^•The same criterion ACTION 
becomes the "given^' in a 
reversal item^ and the criterion 
INPUT is what the ''student wilt' 
produce 








EXAMPLES 


SEE OPPOSITE PAGE 
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EXAMPLES ILLUSTRATIMC PROPERTIES OF TEST ITEM (RESULTS) 
VfHICH CAN BE USED DIAQNGSTICALLY 



EXAMPLES 



ERIC 



INPUT 



rccal ! 



la. i i 



recal I 



lb. « « 



transfer 

Ic.ii 



ACTION 



recal 1 

Id.i 






recal 1 

Ib.i 






transfer 

Ic.i 





2. i 



recal 1 

i a . i i 






reca J 1 

I b . i » 1 






transfer 

1 c . i i i 





2. Ml 



-Nature of errort- made by students 

e.ci. , GIVEN: A6 ^ npufA on Aepan.ate 

H,4X, and lei 
STOPEVT: Taku th^ Aomc 

actions 



e.q. , GIVEN: 



tttt ItemA : la, I, 
la.Aji, and ?a.<xx. 

STUDE^: Taku etc -(j(e 'rent 
acJtioru 2.<>i, 
av\d LiAX] ^oK each 
0^ tiicjn iMtzad 0^ 
tatiina ti\t a owe 
acXlon {l.iYJoK thtm 
aU 

c.^i«» GIVEN: Aa Ajij^itM on Aepcuate 

?a,>ct» and fa.cxx. 

STUPEVT: TafeeA ^ee 

ctcf(j(eA.en/t actconA 

Z.-cc^ onrf l.Ajui] 
but motchei -Cfte/n up 
wJ>ct/i tkz UM.onq^ ^npuXA 
( Ja.-c itiiJdx J •ice) » 
( la. -c>t (i;-c>t/i 2.c.)^ and 
( la.AJui icitli 2*</c) 

e.p.^ GrUEW: Aa ^npu^ on Aepa/iote 
teAt -ctemA--a Ktcatt 
Itcrt [la.i] and a 
Oum^ffCA Itan lla,<A>i) 

STUVENTt A 6tudent mafeeA an 
^aAjox on ont [tht 
voxoviq action—i.ii) 
bat not thz otke.K 



e.a., GIVEN: 



Aa <.npatA on Aegoootc 
cXemA ^0 <,nA tanc^ 
ofi t/ie Aome'lnenBejr'o^ 
a ctaAA ( Ib.li.) 



STUVENT: TafeeA tlie, Aomc 
ffOX botli 0^ them 



F.1.3 

IDCNTIMC4TION 
MATfllX 



CRITERIA rOR rOtNTIfYlMC COWSISTENCY AND tWCOW$ |$TCNCY 
IN ERROR PATTERNS 6h MUlTlKt TESTTTESS 



ERROR PATTERN 


CONSISTENT 
errors 


Inconsistent 
errors 




On multiple te»t items: 

^Pie scone wronn aation is taken 
to successive (multiple) examples 
of IHPU7S from the s<sr\e class 


On multiple test items: 

•i4 different action is taken to 
successive (multiple) excenples 
of INPirrS from the scam class: 

**Ri<7ht and Dronn actions 
matf he found 

• •Different wrong actions 
may be found 



EXAMPLES 
INPUT ACTION 

[73— Q 
4Z1 



he. 

It 



c.^., lofccn qivtn the pKcnoun 

"i/ou" on 4UCC€^A-CVC ttAt 

the ^tudtnt cons<^tcnttt4 
iae4 Che ve^ iow 



e.p., (^en ciivM tht pKonoun 
"i/ou" on 4ucceA4^ve tiMt 
itmA, 

the 4ktiidtnt incoti6i&ttiU; 
AOfieto^e* he oAeA "ate," and 
AonttAjn^ he a&€A "-ca" 



1 
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DETERMINING MOW TO INTERPRET CONSISTENCY/INCONSISTENCV 
ON MULTIPLE TEST ITEMS 



OCCiSlOM 
MATRIX 



ERROR PAHERN 


CONSISTENT 
errors 


INCONSISTENT 
errors 


ACTION 
TO TAKE 


conaiBtency in Brror pattern as: 

^Additional oonfimation of 
diagnoaia 

^Evidence of a ayat^atic error; 
i.e., a definite inaorrect vattem 
has been learned 


incjoncistenru in error pattern ae: 

^Evidence of a variable error; 
i.e., no definite pattern haa 
been learned 



EXAMPLES 


REVfEW EXAMPLE ON PREVIOUS PAGE 


-In vicic ofi tht f^act that tht 

(Wten c^ivtn "twu** 04 a ^ubjtct, 
it -cA concZaded that he ka^ 

foHm of, tht ve^b \*34j:h the pxonoun 


-In vlw 0^ the ^act that the 
student AometcmeA ^OiU '-ca ' and 

it l& cvnctuded that ItU wloh, 
y^axiabte and not huhttmatit. 
He /lOA not aAtocJjOted a t>aKticataA 
((ow 0^ the veAb i^iXh the pfionoun 
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INTCRPRETING ERROR PATTERNS 
INVOLVING ACTIONS ON CRITERION TESTS 



Problems regarding: ACTIONS 

••General 'zat ions 
• •Chai n i ng 


1 

page 


Properties of test results 
whicli car be used diagnostically 


100 


1 nterpret i ng some general 
patterns of results 


102 
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F.3.1 


IDEtiTirYII^G PnOPERTIES OF ERROR PATTERNS r.'VOLVIfJG •'ACTIONS** 


IDENTIFICATION 
MATRIX 






TYPt or 

PROPLRTttS 


act ions 


of actions 




-;;at:a\7 of errors na^Ic ly ct:€.icntc 


"^^atsire of error c nade by ctii^'cntc 


CRITERIA 


* •Cy^ittcd ta^rlKp a): alterr^itive 
artioK ( ore a^^tion atrea^Iy 
learned) 

••Takinj an incorrect alternative 
action 


••Oiittcd exait itini tfie entire 
ncrica of ar.r^cciatroKn 

* ^Qiitted ecchibitinj part of 

tke eerier. 

• •&chihited pai^'tc of the series 

out of ccquence 




-Connistmc:* of errors nade 


^Con^ir.tcncy of errors nade \ 




• •Repetition of error on 
rcteatin-i 


• •Repetition of error on 
rc testing 


^Thcse arc not nutnallu cxi-rlucivc catcgorica; alternative actiorx nay 
involve scqucnccc ^ arul there nay be alternative sequeneac. 


EXAMPLES 


SEE OPPOSITE PAGE 
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F.3.1 


EXAMPLES CONCERNING ERROR PATTERNS IN^^IVINC "ACTION^'* 


EXAMPLES 






TYPE OF 
PROPERTIES 


ALTERflATIVE* 


SEQUENCE* 
o 1 aci > on? 




PROBLEMS REGARDING 
mCTION Cli^ERALI ZATION 


PROBLEMS REGARDING 
CHA 1 N 1 NC 






c.g-t in tJx^cading a ^Ubn 

pKojc<itoK, a 6tudznt: maif 
omit a Loop oJttogt^XhtK; 


EXAMPLES 


/ic catinot M 


iA ht mcuj mafee th^ loop 
oat 0^ 6(Loucnct {and f^oat op 
tilt tivitcuUng) 




dc^4jxlt<0n 0^ tJit concept 
*'antlt^opomo^ plus m" ; 

but fic cannot paKapknoAc 
the dti<jviX4.on o\ ca^mot 
IdtntL^u {e.g.. In a tut) 
tht ^h6tntiat pKopt^tit^ 
oi tht cjonctpt 


(5-3)*, tilt ^tudtnt matf 
^/JUt equate tack nmbcx in 
the poAtntJiueA xathtA than 
iiut Subtracting anxi then 
AquanXng; tixt Stqatnct i^ 
incovicct 




e.g., a 6tudznt con cxmputt a 
•*mttat** ^ox a d^MVUbutton 


e.g., in \tciting a poem ox in 
acting in a ptoM, tht 
Atudtnt may product pa^^agt^s 
Out oH s^utnct 




bat ht Cjonnot compute a 
"mean" bij hand 
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F.3J 

DECISION 
MATRIX 



DETERMINING HOW TO INTERPRET 
GENERAL ERROR PATTERNS COJJCL'RNING ''ACTIONS" 



ERROR PATTERNS 



ACTION 
TO TAKE 



-Error patterns reveal no 
discrimination or generalization 
probleni regarding INPUTS, nor an 
association problem between 
INPUT <ind ACTION 

-On a test item involving an 
example (nev; or old) From the 
same class of inputs, 

an alternative action is omitted 
or an incorrect alternative 
is taken 



INTERPFE7 
as follows 

'Since it lias been established 
that INPUTS and ACTIONS Jiave 
been correctly associated^ 

'Consider this a problem of 
genera li r.atio?^ across ACTIONS 



"Error pat te rns reveal no 
discrimination or generalization 
problem regarding INPUTS, nor an 
association problem between 
INPUT and ACTION 

-On a test item involving an 
examole (new or old) from the 
same class of inputs, 

••All or part of a series of 
associations is omitted 

••The sequence of associations 
i s incorrect 



INTERPRET 
as follows 

'Since it has been established 
that INPUTS and ACTIONS hccoe 
been correctly associated in 
all the separate associations y 

'Consider this a problem in 
chaining a series of associations 
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NOTE 



More detail on interpretation of 
error patterns^ both on practice 
items during instruction and on 
criterion/subcriterion tests, 
are provided in Section J: 
EVALUATE INSTRUCTIONAL MATERIALS. 
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JOB PROCEDURES 





page 


Information to review in 
planning the use of criterion 
or subcriterion test itens 
for diagnostic purposes 


106 


SUMMARY OF PROCEDURES 


107 
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F.3.1 


DETERMINING WHAT INFORMATION TO REVIEW 
IN PLANNING THE LI5F OF CRITERION OR PREPARATORY TF<;TS 
FOR DIAGNOSTIC PURPOSED 


DECISION 
MATRIX 








For each CRITERION BEHAVIOR 


Are there likely to be 


INFORMATION 
NEEDED 


are there likely to be 
learni ng fa3 lures 

! nvol ved in lea rn i no some Dart 
or all of the criterion behavior? 


-Enough criterion and subcri terion 
test Items 

^V^r i i^^d ont^Lioh erf f^rif^n ;)nd 

subcriterion test items 

to diagnose precise type 
of learning failure? 




-Re JieD the set of "statement of 
objectives'^ associated with each 
CT-'tevion hehaoiov for: 


-Review the entire set of 
criterion a>id subcriterion test 
items for coverage of: 


ACTION 
TJ TAKE 


• * Vrepavatory objectives 
identifying component skills 
luhich are likely to pose 
laaxming difficulties 


••All the INPUT classes and 
ACTION classes likely to be 
involved in learning failures 

••Recall/transfer items relating 
to the INPUT/ACTION classes 

••Multiple examples within each 
class 

••Multiple instances of the same 
example (a retest item) 
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F.3.1 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN 
PLANNING THE USE OF CRITERION OR PREPARATORY TEST ITEMS 
FOR DIAGNOSTIC PURPOSES 



#1 



n 



For EACH CRITERION BEHAVIOR 
REVIEW: 

a. The subcriterion 
objective which is 
associated with it 
and which identifies 
potential learning 
problems 

b. The criterion and 
subcriterion tests for 
adequacy of coverage 
of potential learning 
problems (and likely 
fai lures) 



DECIDE 

a.l To prepare diagnostic 
items--\vhen coverage 
by criterion or 
subcri terion i tems is 
insufficient for 
diagnostic purposes 

a. 2 Not to prepare 

d i agnost i c i tems --when 
coverage by criterion 
or subcriterion items 
is sufficient for 
diagnostic purposes 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


Diagnostic probes to supplement test procedures as a 
means of diagnosing learning failures and their 
probable causes. 


WHAT YOU WILL 
WORK FROM 


(1) Statements of objectives 

(2) Task and learning analyses Indicating what the 
likelihood Is of learning failures occurring 

(3) Criterion test items 


WHAT YOU WILL 
DO 


(1) Review above Information and determine whether 
tests are tikely to be sufficiently dtannostic of 
learning failures and the causes for then« 

(2) Develop diagnostic probes when necessary* 


FORMS YOU WILL 
USE 


None 



ERIC 



108 



DESCRIPTION OF Sub STEP 



INPUT 



Statement of objectives 
-^ 

task analysis results 

learning analysis 
results 



criterion test itens 



IV 



F.3.2 



ACTION 



Develop diagnostic 
probes vhen necessary 



OUTPUT 



PROBES 
To diagnose type of 
learn 'ng failure and 
sOTorces of the failure 



vi 



Job Aid I ontcnts 



CRITERIA FOR 



IDENTIFYING INPUTS 


ACTION TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 


-MATRIX: Using 
probes regarding 
criterion tests 
vs. diagnostic 


-MATRIX: Types of 
probes to 

use . . . n6-;2t 

-MATRIX: Developing 
probes . . 123-128 


-MATRIX: Adequacy 
of probes which 
have been 

developed ... 129 


SUMMARY OF 
PROCEDURES . . 128 


Required MaltTials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BIANK FORMS 


Oeterminat ion of 
i ikely need of 
diagnost i c in format lor 


F.3.1 


Forms associated with 
cr i terion behavior 
(carried forward frorrj 


F.3.1 























































Sub- STEP 



I F.3.2 I 



INPUT 



Statement of objectives 

task analysis results 

learning analysis 
results 

criterion test items 

iv 



JOB DIAGRAM 



ACTION 



Develop diagnostic 
probes when necessary 



OUTPUT 



PROBES 
To diagnose type of 
learning failure and 
sources of the failure 



Anticipated 
DISCRIMINATION 
problems 



iv.a 



Develop specific probes 
to determine this type 
of problem and I ts 
source 



v.a 



QUESTIONS 

■Re: INPUTS and their 
properties 

-Re: INPUT examples 
requiring Indication of 
••sam«'* or ••different'* 

Re: Examples varied 
systematical )y for 
simi lari ty 

vi.a 



Anticipated 
GENERALIZATION 
problems 



iv.b 



Develop specific probes 
to determine this type 
of problem and its 
source 



v.b 



QUESTIONS 

hRe: INPUTS and their 
properties 

-Re: INPUT examples 
requiring indication o 
••same'^ or ••dl f ferent*' 

Re: Examples varied 
systematically for 
simi larlty 

vl.b 



Anticipated 
ASSOCIATION 
problems 



iv.c 



Develop specific probes 
to determine this type 
of problem and its 
source 



v»c 



QUESTIONS 

-Re: INPUTS and ACTION 

-Reverse direction of 
performance 

-Correctness of pairing 



vt .c 



TWO APPROACHES TO OBTAINING 
SPECIFIC DIAGNOSTIC INFORMATION 





page 


Two ways to obtain 

specific diagnostic Information 


112 


When tO' use each of the two 
diagnostic approaches 


113 
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F.3.2 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING TWO WAYS TO 08TAIN 
SPECIFIC DIAGNOSTiC INFORMATION 



TWO METHODS 


Preparing and using 
DIAGNOSTIC TEST ITEHS 


Preparing and using 
PROBING QUESTIONS 
with students 
concerning their performance 
on criterion and preparatory 
test items 


CRITERIA 


'Test items which are designed 
specifically to identify 
failures: 


^Questions used in interviews 
with students following their 
performance (and c emission of 
errors on) criterion and 
preparatory tests 

'Questions which are designed 
specifically to identify 
failures: 




Regarding: INPUTS 

• •Discriminations 

• •Generalizations 




Regarding: INPUTS and ACTIONS 
• •Associations 
• •Chains 




ReguMing : ACTIONS 
••Generalizations 



ERIC 



112 



F.3.2 

DECISION 
MATRIX 



DETERHINING WHEN TO USt EACH OF THE TVO WAYS 
TO OBTAIN SPECIFIC DIAGNOSTIC INFORMATION* 



CONDITIONS 


-A full range of multiple c*-iterion 
(or preparatory) test Items have 
not been developed, thus not 
allowing patterns of results to 
be analyzed 

-Multiple test item patterns are 
di rf icult to interpret 


-Specific preparatory objectives 
hove been stated 

which identify specific cotmponent 
skills whose required acquisition 
is likely to pose learning 
difficulties 


ACTION 
TO TAKE 


Plan to intcrvi^'j students and 
probe for sources of learning 
failure 


Plan to develop spc<r'il diagnostic 
tes ts 


METHODS 


See page 1 15 


See page 131 



^Both methods are likely to be used during development of instructional 
materials. In the routine administration of a fully developed and 
validated instructional program, only diagnostic tests are likely to b^ 
used. At that time their us^ is concerned wvth the specific proble^^^ 
of individual students rather than with problems about the adeq^^u^y of 
the instructional materials. 
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FOLLOWING ADMINISTRATION OF CRITERION TESTS: 
PROBING FOR SOURCES OF LEARNING FAILURES* 





page 


Priorities in order of question 
types used 


116 


Two types of errors: (1) omissions 
and (2) wrong answers, requiring 
different interpretations and 
probing 


117 


Interpreting replies to probes 
indicating: (I) something not 
learned, OR (2) something incorrectly 
learned 


118 


Probes to determine %rfhethcr: 

(1) something not learned, OR (2) 

something Incorrectly learned 


120 


Other probing procedures to use to 
Identify specific types of learning 
fai lures 


121 



^Similar probing can be performed with practi^^ itema 
during infomal tryouta of inatiw^tional materials; 
a one-to-one try out and interview eeaeion involving 
an administrator and a single student can be used to 
identify sources of failures on criterion practic e 
items which are a part of the instructional program. 
See SECTION J. Section J also provides detailed 
considerations in interpreting errors on single test 
items, responses to test probes, and patterns of 
errors on multiple test items. 



115 



F.3.2 

DECISION 
MATRIX 



DETERMINlKfi PRIORITIES IN THE ORDER IN WHICH' 
DIFFERENT TYPES OF PROBING QUESTIONS SHOULD BE ASKED 



PRIORITIES 



ACTION 
TO TAKE 



FIRST 



-Ask open-ended 
questions which do not 
identify or suggest 
answers he tnight give 



-Do not call attention 
either to the INPUTS 
or to the ACTIONS for 
him to focus on 

'Do not call attention 
to specific properties 
of INPUTS or of 
ACTIONS to focus on 



If f i rst answers 
provide indeterminate 
resul ts 

SECOND 



-Ask directed questions 
which identify what 
the student should 
consider when trying 
to assess what went 
wrong 

^ Call attention 
generally either to 
the INPUTS or to the 
ACTIONS for him to 
focus on, BUT 

'Do not call attention 
to specific properties 
of INPUTS or of 
ACTIONS to focus on 



If second answers still 
provide indeterminate 
resul ts 

THIRD 



-Ask directed questions 



'Call attention to 
INPUTS or to ACTIONS 



'Call attention to 
specific properties 
for student to focus 
on 



EXAMPLES 



ERIC 



"Why did you 



take this ACTION 



e./}. , "W/iy did you 



■"What Is there about 
this that made 



INPUT 



you 



V 



take this ACTION 



e.g. 
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"W/iat pKopVitxj 
OK attnZbatc of^ 
thJU lojOii madz 
ifou cZa^^l^tf It 



^'Vhlch of these 

of the 

properties 

made you 

INPUTS 

^7" 

take this ACTION 

e.g., "Which tka^z 

^^cze, 6hapz , ok 
coloK madt yon 
makz tkih 
pajitiaxtdK 



F.3.2 

DECISION 
MATRIX 



TYPES OF 
ERRORS 



ACTION 
TC TAKE 
(INTERPRET) 



DETERMINING HOW TO INTERPRET TWO TYPES OF ERRORS 



NO action 
is taken on a test Item 

• •An omission 

**No answer given 



On RECALL items 

' Moet likelu indicative of: 

••A failure to acquire or to 
retain a dieorimination, a 
generalization, or an 
aaaooiation in the firat place: 
learning did not occur 

-Leee likelu (but poaaibly) 
indicative of: 

••An acquiaition of an INCORRECT 
diacrimination, generalisation, 
or aaaooiation; i.e., aamething 
wrong ia learned 



on TRANSFER itema 

' Moat likely indicative of: 

• •A failure to aoauif^ cl 
generalization 



' Leaa likelif (but poaaibly) 

• • Interpretatiana provided above 
for RECALL itema 



A WRONG action 
is taken on a test item 

• 'A wrong answer 
is given 



On RECALL itema 

' Moat likely indicative of: 

• •An acquiaition of an INCORRECT 
diacrimination, generalization, 
or aaaociation; i.e.^ aamething 
wrong ia learned 

' Leaa likelu (but poaaibly) 
indicative of : 

••A failure to acquire or to 
retain a diacrimination, a 
generalization, or an 
aaaociation in the firat ylace; 
learning did not occur 

on TRANSFER itema 

' Moat likely indicative of: 

••A wrong generalization haa 
been made; or 

• •A wrong diacrimination has 
been made 

' Leaa likely (but poaaibly) 

••Interpretations provided above 
for RECALL itema 
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F.3.2 
IDENTIFICATION 
MATRIK 



CRITERIA FOR lOENTIFYIMG WHETHER LEARNING DID NOT OCCUR 
OR WHETHER INCORRECT LEARNING TOOK PLACE 



TYPE OF 
PROBLEM 



Learning 
has NOT taken place 



INCORRECT learning 
has taken place 



CRITERIA 



HIGHLY likely when: 

'On another RECALL teat item, 
cm anauer is omitted 

AND 

'In reeponae to probea, he aaya 
one or more of the following: 

••J don^t know the ansoer 

••J don't know why I left it out 

'In reply to probea, atudent doea 
any of the following: 

••Doea not attribute propertiea 
to INPUTS 

••Doea not indicate that he ia 
oonfuaed about INPUTS (or their 
propertiea) 

••Doea not identify (aeveral) 
poaaible anewera 



HIGHLI likely when: 

'On another RECALL or TRANSFER 
item, makea the acme error 

'In reply to probea, the atudent 
doea one or more of the following: 

••Attributea propertiea to 
INPUTS incorrectly 

••Indicatea general or apecific 
confuaion about INPUTS (or 
their propertiea) ; he pay a 
attention to irrelevant 
propertiea 

••Identifiea possible answers 
but indicatea he doea not 
knew which ia the correct one 



EXAMPLES 
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SEE OPPOSITE PAGE 



F.3.2 

EXAMPLES 



EXAMPLES ILLUSTRATINC INTERPRETATION OF STUDENT RESPONSES 
TO ADDITIONAL TEST ITEHS OR TO PROBES 



ORIGINAL 
TEST I TEH 



IVEN: An old (RECALL) example of a "fJrst class" lever (i.e., fulcrum 
between effort force and resistance force) 

STUDENT IS 

REQUIRED TO: Identify the ''class'' the lever belongs to 



Student OMITS answer 



Student gives WRONG answer 



EXAMPLES 



On another test Item with 
another example belonging 
to the same INPUT CLASS, 

i.e., ttvvu i/ifhich havt 
tht iulcAum btOattn tht 
mo/tt ioKct and tht 

The student also giv^s no answer [-The studrnt gives the same wrong 

a.is%ifer 

e.g., c/ju^Hie^ It <u a thiKd 
cltu^ ttvtn. 



On probes the 
one or more of 



student does 
the following: 



Says: 

e.g. , I don't knou) 

e.g. , I don't knout ufhy 

e.a.. I don't hou) you tttt 
i/utUch typt It 

Gives none of the responses in 
the adjacent column 



^Attributes properties to INPUT 
incorrectly: 

e.g., ^ay^ tht "Kthi&tanct ioKct" 
Ia btJb/ottn tht ti^o^ ioKCt 
and tht iulcAam 

-Attends to irrelevant properties 

e.g., 6ayA ht gave hu onAuDZA 
btc/uut oi tht "doumkxAd 
dOttction" 0^ tht KtAiAtanct 

ioKCJt 

identifies all the possible 
answers but doesn't know which 
are requi red 

e.g., Idtntt^lju pfioptAtLtA o^ 
ItvtM coHJitdtJbi and thtOt 
diiitAtnctA, but dotAn't 
knou) uJhich tabtl gotA uUth 
wkLch 



LIKELY PROBLEM 



erIc" 



Learning has NOT occurred 



Incorrect learning has occurred 



F.3.2 

DECISION 
MATRIX 



TYPES OF 

SUSPECTED 

PROBLEMS 



OETERHINING HOW TO USE ADDITIONAL TYPES C; PROBES 
TO IDENTIFY SPECIFIC. SUSPECTED LEARNING FAILURES 



Incorrect 
DISCRIMINATIONS 



Incorrect 
GENERALIZATIONS 



Incorrect 
ASSOCIATIONS 



ACTION 
TO TAKE 



'Pr€B0nt additional 
sxcmplMB from a 
different INPUT oIobb 
wkioh ie represented 
in the original test 
item tmd ask: 

••Are these the same 
or different? 

OR 

••Should these be 
handled or treated 
the seme way or in 
different ways? 

'If examples vary 
dimensicmallff . present 
poire of INPlfT exarnplei 
(using the original 
excnple as one member) 
whi(^ start far apart 
and gradually are made 
more similar 

••Detemine when and 
where discrimination 
breaks down 



'Present additional 
excmiples from the sme 
INPUT class whieh Ts 
represented in the 
original test item 
and ask: 

• •Are these the same 
or different? 

OR 

••Should these be 
handled or treated 
the sme way or in 
different ways? 

'If exanples vary 
dimensionallys present 
pairs of INPUT excmples 
(using the original 
excnple as one member) 
which start close 
together and gradually 
are made more 
dissimilar 

••Detemine when and 
where generalization 
breaks down 



'If behavior has been 
taught in two 
directions : 
INPUT ACTION and 
ACTION INPUT: 

••Reverse a test 
question and 
detemine whether 
same mismatch occurs 



'Identify for the 
student all the classmB 
of inputs and all the 
actions that can be 
taken and then ask: 



•Which 

ACTION 
with which 



goes 



INPUT 



•This is haj I pair 
them up. Am I right 
or wrong? 
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SECTIOM J of the HANDBOOK provider 
detailsd aomidsratian of uzi/t to 
interpret Mtudmnt rm^potiBmB to 
prob€B. If alBo dmalB with tnt«r- 
pretation of errors on Bingle and 
on multiple test items, ^Jnt«r- 
preuxtion^ is presented at that 
point in the development proeeee 
at which it haa practical implica" 
tione; i.e.^ it provides the basis 
for detsmining how to revise 
instructional materials. 
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JOB PROCEDURES 



page 



TYPES OF INFORMATION YOU WISH 
TO OBTAIN FROM THE USE OF PROBtS 


]2k 


DETERMINING WHETHER TO DEVELOP 
PROBING PROCEDURES 


125 


Two points during the development 
process when probes may be 
selected and/or prepared 


126 


Information to review during 
TEST DEVELOPMENT only in order 
to identify probes which are 
needed 


127 


SUMMARY OF PROCEDURES 


128 


Adequacy of probes 


129 
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F.3.2 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDENTIFYING TYPES OF INFORMATION 
YOU WISH TO OBTAIN BY USING PROBING QUESTIONS AND PROCEDURES* 



TYPES 
OF INFORMATION 
TO BE 

OBTAINED 



TYPE OF 
FAILURE 
WHICH HAS OCCURRED 



CRITERIA 



VARIABLES 
ACCOUNTING FOR 

t;sE failure 



'Has no^ learning taken place 



'Has incorreat leaxming taken 
place involving: 

• • Discrvminatiana 

• •Generalizations 

• •Associations 



-The sources of "no learning 
occurring" lie primarily in the 
nature of the instructional 
materials, for example, in the 
amount of review provided or in 
the use of ;^ransfer practice 
items, etc. Probes generally 
are not designed to identify 
this type of information. 

Responses to probes mxy, on the 
other hand, identify sections 
of the instructional materials 
which have confused students 

'Have learning failures occurred 
due to: 

' 'Student attention to irrelevant 
properties of INPUTS 

••Student confusion created by 
similarity /dissimilarity of 
INPUT examples 

••Num^^r of properties of INPUTS 
or • ; ACTIONS to attend to 



^WUILE OTHER DIAGNOSTIC PROCEDURES INVOLVING THE ANALYSIS OF TEST RESULTS 
CAN LEAD TO AN IDENTIFICATION OF THE TYPE OF LEARNING FAILURE, IT IS ONLY 
THE PROBING PROCEDURES WHICH DIRECTLY AND E/^fPIRICALLY IDENTIFY THB^ REASONS 
FOR THE FAILURE. 



ERLC 



12^ 



F.3.2 

DECISION 
MATRIX 



CONDITIONS 



OETERHINING WHETHER TO DEVlLOP PROBING PROCEDURES 



-Statement of objectives includes 
subcri:erion objectives 
(indicating probability of 
learning difficulties) 

-Associated learning analyses 
also reveal probable learning 
di f f icul ties 

-The number of criterion test items 
is relatively s mal ^ (thus making 
Interpretat ion of test results 
more di ff icult) 



-Statement of objectives does NOT 
include subcriterion 
objectives 

-Associated learning analyses 
do NOT reveal probable learning 
difficulties 

-The number of criterion test items 
Is relatively large (thus making 
interpretation of test results 
easier) 



ACT I on 
TO TAKE 



Definitely consider deveJvping 
and using probing questions 
and/or procedures 



Give some consideration to 
developing and using probing 
questions and/or procedures^ 



^The probing procedure is, as already pointed out, the stalest direct way 
of identifying tite factors_ leading to each type of failure. 
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F.3,2 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDEMTIFYING TWO POINTS IN THE DEVELOPMENT PROCESS 
WHEN PROBES ARE SELECTED AND/OR PREPARED 



TIME 
iK THE 
DEVELOPMENT 
PROCESS 



FIRST* 



SECOND 



^During the deve lopnen t of 
criterion and CMcntcxn-oii teat 
itcns, i.e., during TASK F: 
"DEVELOP DIAGNOSTIC AND 
EVALUATIVE TESTS" 

^At this point in time, prepara-^ 
tion of probcG ie bcuscd on 
anticipat'^d lca}'ning failures 



'During the evaluation of already 
prepared instructional n.aterials , 
i.e., during TASK J: "EVALUATE 
INSTRUCTIONAL MATERIAT£'* 

'At this point in time, probes 
are changed and/or new probes 
are developed based on test 
errors which students have 
actual lu made in test items 



^Performance of Sub-STEP F.Z.2 at this stage of the development process 
(i.e., during development of TESTS) can be considered "optional.^* 
Development of probes can^ without loss, be postponed until test results 
have becanc available and you wish to probe for sources of specific 
learning failures you suspect. 
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DETERMINING WHAT JNFORMATION TO REVIEW 
AS A BASIS FOR DEVELOPING PROBES CONCERNING STUDENT ERRORS 
ON CRITERION AND PREPARATORY TEST ITEMS 



UECISION 
MATRIX 



INFORMATiON 
NEEDED 


Are tlicre 
likely to be 
learning fai lures? 


What kinds of 
informat ion 
should the probes 
be des iqned to get? 


Is there 
1 ikely to be 
difficulty 
interpreting 
results on 
cri terion tests? 


ACTION 
TO TAKE 


REVIEW: 

^Statement of 
objectives on 

D.2(l) for 
presence of 

••Prcpara torj 
ohJectivcG 

^Collection of all 
learning analysis 
forms [F0EI4 A. 5(4) or 
(11)] accociatad with 
the criterion behavior 
for identification of 
LVKe LXf uecuTivng 
failures 


REVIEW: 

'Task analysis forms 
[FORM A. 5(4) or (11)] 
in order to identify: 

••The criterion 
inputs and actions 

••Their properties 

-Collection of all 
learning analysis 
forms \FOm A.5(4) or 
(11)] arr.oaiatrd with 
the criterion behaoior 
for identification of: 

••What type(s) of 
learning failures 
are likely to occur 

••Potential sources 
of the difficulties 


REVIEW: 

'Criterion test item 
on F0R14 F.2(l) in 
order to identify. 

••The number of 
criterion and 
subcriterion test 
items devoted to 
the criterion 
behavior 
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F.3.2 



ILLUSTRATION SU^HARIZING PROCEDURES INVOLVED IN DECIDING 
WHETHER TO DEVELOP AND IN DEVELOPING DIAGNOSTIC PROBING PROCEDURES 



II 



#2 



REVIEW 

a. Statement of objectives 
on FORM D.2{I) for 
presence of subcriterlon 
objectives 

b. Identification of types 
of potential learning 
difficulties on set of 
FORMS A.SC*) or (ll) 

c. Identification of relevant 
properties of INPUTS and 
ACTIONS in task analysis 
on set of FORMS A.SC*) or 
(II) 

d. Number of criterion test 
Items and subcrfterion 
test items associated 
with the criterion 
behavior on FORM F.2(l) 



DECIDE 

Whether to develop probes: 
a. I How 

a. 2 Later when actual 

test results are in 

DEVELOP 

Probing questions or 
probing procedures 



ERIC 
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F.3.2 

STAK'DAROS 
MATRIX 



PROPERTIES FOR ASSESSING THE ADEQUACY 
OF THE PROBES WHICH HAVE BEEN DEVELOPED 



PROPERTIES 



Probes avai lable 
which arc GRADED 
for amount of 
CUING provided 
to student 



RELEVANCE 



APPROPRIATENESS 
FOR AUDIENCE 



CRITERIA 



'Types of probes uhich 
should be available: 

••Fully open-ended 

••Less open-ended 

••Highly directive 

'Planned order of 
usage : 

• •Same order as above 



'Probes should be 
geared to the 
identification of: 

• •Suspected or 
anticipated types 
of learning failures 

••Factors which are 
possible reasons 
for the anticipated 
failures 



^Probes should provide 
tasks which students 
can easily perform 

'Wording of probes 
should be: 

• •Brief 

••Clear 

• •In language 
audience can 
unders tand 



* 0 
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Pr.Lv^lEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A diagnostic test appropriate to anticipated types 
of leamirjg failures • 


WHAT YOU WILL 
WORK FROM 


(1) Statements of objectives. 

(2) Task analysis and learning analysis results 
Identifying likelihood of learning failures* 

(3) Criterion test Items. 


WHAT YOU WILL 
DO 


(J) Dftvelop diagnostic test Items geared to anticipated 
types of learning failures. 


FORMS YOU WILL 
USE 


FORM F.2(l) for developing diagnostic test Item:^. 
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DESCRIPTION OFSub-STEP 



F.3.3 



INPUT 



ACTION 



OUTPUT 



Statement of objectives 
+ 

task analysis results 
+ 

learning analysis 
results 

+ 

criterion test items 

vii 



Develop diagnostic 
test items appropriate 
to anticipated types 
of learning "f^ailures 



viii 



DIAGNOSTIC TEST ITEM 
to diagnose type of 
learning failure 



ix 



Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 



-MATRIX: Types 


-MATRIX; 




of diagnostic 


Priorities in 




test 


order of 




items ... 133-137 


administering 






test i terns . . 






-MATRIX: 






Information 






to review as 






a basis for 






developing 






d 1 agnost i c 




I 


test i tems * . 





-MATRIX: Adequacy 
of diagnostic 
test items . . . 



1^3 



F.2(l) 

SUMMARY OF 
PROCEDURES 



\k2 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Determination of 
I Ikely need of diag- 
nostic information 


F*3.2 


Forms associated with 
criterion behavior 
(carried forward from! 


F,3-2 


FORM F.2(]) 










































C 1 
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Sub- STEP I F,3.3 I 



JOB DIAGRAM 



INPUT 



ACTION 



OUTPUT 



Statement of objectives 

task ajialysis results 

learning analysis 
results 

criterion test items vi 



Develop diagnostic 
test items appropriate 
to anticipate I types 
of learning failures 



vlii 



DIAGNOSTIC TFST ITH4 
to diagnose type of 
leco^ning failure 



ix 



ANTICIPATED PROBLEMS 



DISCRIMINATIONS 
anx)ng INPUTS 



VI t .a 



Develop diagnostic 
test Items appropriate 
:o anticipated problem 



viil.a 



GENERALIZATIONS 
among INPUTS 



vii .b 



Develop diagnostic 
test items appropriate 
to anticipated problem 



vit i.b 



ASSOCIATIONS 
between 
INPUTS and ACTIONS 



vi r.c 



Develop diagnostic 
test items appropriate 
to anticipated problem 



viil.c 



GENERALIZATIONS 
across ACTIONS 



vILd 



Develop diagnostic 
test Items appropriate 
to anticipated problem 



viii.d 



TEST ITEMS INCLUDE 



Two or more INPUT 
examples from same 
and/or d.fferent INPUT 
classes; testlrj for 
abi 1 1 ty to see 
difference between 



classes of INPUTS* 



Ix.a 



Two or more INPUT 
examples from same 
and/or different INPUT 
classes: testing for 
abi I i ty to see 
simi lari ty within a 



class of INPUTS 



Two or more pairings of 
INPUTS AND ACTIONS, 
some of which are 
correct and some 
Incorrect: testing for 
abi lity to identify 
correct pairings 



ix.c 



Two or more ACTION 
examples from same 
and/or di fferent 
classes: testing for 
abi 1 i ty to see 
simi lari ty of ACTIONS 
thin a class of 



ACTIONS 



ix*d 



^When it is pairs of ACTIONS that are presented, the sPjdent can be tested for hie ability 
see differences amono actions (which may be a problem in some learning situations) . 
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BACKGROUND INFORMATION 





page 


Types of Information results 
on diagnostic test items 
should reveal 


' 13^ 


Testing for bludent ability 
to see "sameness" or 
"differences" 


135 


Testing for student ability 
to see "correctness" or 
"incorrectness" 


135 


Types of recognition Items 
useful in diagnosis 


136 


Developing recognition Items 
for diagnosing specific 
types of problems 


137 
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F.3.3 

DECtSIOlM 
MATRIX 



TESTING FOR 
STUDENT ABILITY 



DETERMINING HOW TO TEST FOR STUDENT ABILITY 
TO ''SEE*^ SIMILARITY OR TO "'SEE'' DIFFERENCES 



To see 
SIMILARITY 



To see 
DIFFERENCE 



ACTION 
TO TAKE 



•-Use RECOGNITION test items 

--When testing far ability to see svnilarity or differences 
arnong INPUTS 

••Do not require as the answer an identification of the action 
associated with one or more of the inputs 

^•g*s do not ask: "Do you say 'J5' for each of these nouns 
or for both of them?" 

• •Vo^ require as the answer a neutral identification of 
sameness or difference 

e.g., "Are these the same or different?*' 

e.g., "Do you treat these the same way or in different ways?" 

"When testing for ability to see similarity or differences 
among ACTIONS 

••Do not require an identification of inputs 

••Do require (simply) whether actions can or cannot be used 
tnterchangeab ly 

e.g., "Are these the seme or different?" 
^^g.j "Are these equivalent?" 

e.g., "Would you use these under the same circumstances or 
different circums tances ? " 



F.3,3 

DECISION 
MATRIX 



DETERMINING HOW TO ALLOW STUDENT 
TO INDICATE "CORRECT" VS. "INCORRECT" 



TESTING FOR 


To see 


lo see 


STUDENT ABII ITY 


CORRECTNESS 


INCORRECTNESS 




-Use RECOGNITION test items 




ACTION 






TO TAKE 


-When testing for ability to see oorreatnesa or inoorreotnees 




of asso(yiations between IIWUTS and ACTIONS: 




••Simply require an indication of correctness vs. incorrectness 



ERIC 



135 



F*3*3 

IDENTIFICATION 
?i/IATR1X 



CRITERIA FOR IDENTIFYING TYPES OF RECOGNITION ITEMS 
USEFUL IN DIAGNOSING ERRORS 



TYPES 
OF ITEMS 



CRITERIA 



as 



Us 1 ng 
PAIRS OF EXAMPLES 
the ''GIVEN'^ in a test 



i tern 



Re: INPUTS 

-Pairs of examples from tuo 
different clasces are used to 
determine ability to see 
differences 

••As many pairings of examples 
from U^o classes as there are 
combinations 

e.g*^ if there are three 

classes, three pairs 
would be tested 

e.g., if there are four classes, 
six pairs would he testedf^ 

'Pairs of examples from the same 
class to determine the ability 
to see similarities 

••At least one pair representing 
each class 



Re: ACTIONS 

'Pairs of cxcmples of actions 
from different classes 

'Pairs of examples of actions 
from same class 



Re: INPUTS and ACTIONS 

'Pairs of correct and incorrect 
associations 



Us ing 
MULTIPLE EXAMPLES 
as the *'G!VEN*' In a test 



i tern 



lie: INPUTS or ACTIONS 

-Lieut collections of excenples 

••Aa^v <,^.Z be from same class 

••May all be from different 
classes 

• •May be a mixture 

-Student must identify: 

••Whether examples are the seme 
or different 

OR 

••Which examples are the same 
or different 



Re: INPUTS AND^ ACTIONS 

-List or collection of examples 
of association 

••May be all wrong 

••May he all right 

' •May be a mixture 

-Student must identify: 

••Whether they are correct or 
incorrect associations 

••Which are correct and which 
are incorrect 



*e.g., FOUR CLASSES (A, B, C, O) 



EXAMPLES 



•AC 
•AP 



8C 
SI? 
CV 
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ERLC 



JOB PROCEDURES 





page 


Priorities in the order in which 
diagnostic type tests are 
administered 


UO 


Information to review in planning 
the preparation of diagnostic 
test Items 


HI 


SUMMARY OF PROCEDURES 




Adequacy of diagnostic test. Items 


1^3 



M« in the case of the development of probes^ diavwatic 
teat itema can be developed at the time other types of 
teat item are being developed. Their development can 
be postponed until Taak J when instructional materials 
are being tried out and errors on instructional materials 
and on criterion tests have becane available* If these 
results (and/or results of probing procedures) do not 
uncpnbiguoualy identify the specific type of learning 
failure, diagnostic tests may be developed which are 
specific to the residual problems in diagnosing learning 
failures. 
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F.3.3 



DETERMINING THE APPROPRIATE ORDER IN WHICH TO ADMINISTER THE DIFFERENT 
TYPES OF DIAGNOSTIC TESTS (Either In STEP F.k or in TASK J) 



DECISION 

MATRIX I: 2 3 



CONDITIONS 


No diagnostic 
tests have yet 
been administered 


Results of first Items 
administered reveal 
NO difficulties 
in discriminations 
or generalizations 
regarding INPUTS 


Results of first items 
administered reveal 
NO difficulties 
regarding ACTION 
general ization 
or in discriminations 
among ACTIONS 


ACTION 
TO TAKE 


FIRST 

-Administer diagnostic 
items testing for: 

• •Discriminations 

• •Generalizations 

regarding INPUTS 


SECOND 

'Administer diagnostic 
items testing far: 

• Action generaliza-- 
tions 

• •Discriminations 
among ACTIONS 
(optional) 


THIRD^ 

-Administer diagnostic 
items testing for: 

••Associations 
between INPUTS and 
ACTIONS 



*8y postponing testing of ASSOCIATIONS until last, it is possible to make 
a differential diagnosis of an "association" problem* If administered 
earlier, items involving both INPUTS and ACTIONS would produce results 
which are indeterminate as to the type of learning failure. 



DECISION 
MATRIX 



INFORMATION 
NEEDED 



DETERHINING WHAT INFORMATION TO REVIEW 
AS A BASIS FOR DEVELOPING DIAGNOSTIC TEST ITEMS CONCERNING 
STUDENT ERRORS ON CRITERION AND PREPARATORY TEST ITEMS 



Arc there 
MkcIy to be 
learning fai lures? 



What kinds of 
information should 
the diagnostic items 
be desfgned to get? 



Is there 
likely to be 
di f f icul ty 
interpret ing 
results on 
criterion tests? 



ACTION 
TO TAKE 



REVIEW : 

'Statement of 
objectives on 
FOmi D.2(l) for 
presence of 

• .Subai^lterian 
objectti^eo 



'Collection of ail 
learning analyeie 
foms [FORM A. 5(4) or 
(11 J] ascociated with 
the criterion behaoior 
for identification of 
likely learning 
failuree 



REVIEW : 

'Task analysis forms 
[FORM A.5(4) or (21)] 
in order to identify: 

• *The criterion 
inputs and actions 

'•Their properties 



'Collection of all 
learning analysis 
forms [FORM A. 5(4) or 
(11)] associated with 
the criterion behavior 
for identification of: 

• •What type(e) of 

learning failhur^es 
are likely to occur 

* •Potential sources 

of the difficulties 



REVIEW : 

'Criterion test item 
on FORM F.2(l) in 
order to identify: 

• •The m^nber of 
criterion and 
subcriterion test 
items devoted to 
the criterion 
behavior 
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ILLUSTRATION SUMMARIZING PROCEDURES 
IfWOLVEO IN PREPARING DIAGNOSTIC TEST ITEHS 



#1 



REVIEW 

a. Statement of objectives 
on FORH 0-2(1) for 
presence of 

subcritcrion objectives 

b. Identification of types 
of potential learning 
difficulties on set of 
FORMS A.5(^) or (II) 

c. Identification of relevant 
properties of INPUTS and 
ACTIONS in task analysis 
on set of FORMS A.5(^) or 
(11) 

d. Hunter of criterion test 
items and subcriterion 
test items associated 
with the criterion 
behavior on FORM F,2(l) 



#2 

DECIDE 

a. Whether to develop 

diagnostic test items: 

a. I Now 

a.2 Later when actual 

test results are in 



DEVELOP 

Diagnostic test questions: 
Use FORM F.2(l) to record 
each test iteni 
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F.3.3 

STANOAAM 
MATMIX 



CHITERIA FOR ASSESSING THc ADEQUACY 
or OIMNOSTIC TEST ITEHS 



MOPEHTIES 



RELEVAilCE 
and COVERAGE 



AVOiOANCE 
or M)ll*0 1 rr ERENTIAL 
RESULTS 



rtAWIED ORDER 
OF ADH III i STRATI ON 



CRITERIA 



-Tot itmm •hould b#: 

'Plam^d ordMT of 

••Sam ord^r a» abov€ 

'Sampling of t#«t itmm 
•hould b0 miffioimt 
to tB9t for all thm 
(mticipatsd probUmB 



•hould WOT rmquira 
idontifioaiion of tho 
ACTTOtf a990oiatsd 
iKth an IWWT * 

^Studant ra9pona99 
Bhould only raquir0 
idontifioation of 

• •Sma/diffaroHt 

• •Corraot/inaorr^et 



-799t it0m9 aaquanoad 
to ta9t for probUmB 
in thi9 ordar: 

FIRST: Diaerimina" 
Horn and 
ganaraliMa* 
tiona ragard-- 

ing rwPUTS 

SEC0J9D: ACTIOH 

ganaraliMa-' 
tiona^ 

THIRD: A99oaiationa 

haty^aan iHPUTS 
md ACTI09S 



^yhmre c^lieabls: Di9arim.nationa rtgaxding ACTIOHS whiah may b9 oonfuBod 
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EXAHTICS ILLUSTMTING OIAGHOSTIC UCT ITCHS CCARCD TO rtOVIDC 
IMfO^ATtOfi AtOUT SfCCIFtC mc$ Of AMTICIMTCO LEAMIMG TAILOUCS 



TYPE OF 
ANTIC I FATED 
FHOILEH 



-OISCHIMINATIOHS 
•GENERALIZATIONS 



t.g., TEST ITEM 

III IXI . IV) 

If) (X) ♦ {•/) 

Should tktM be VuxLttd: 

^Ifi tkt iomc M*a4f 

In diiitKtnt noojL!^ 

Di«9nosttS • pottlbU 

^roblM. Faifure to 
9«Mr«HM across two 
sott of symbolt which 
signify tho %9m thing. 



€.4.. TEST ITEM 

(I) (X) - IV) 
12) fX) - (-y) 
Sfcoatd £keif be tJuajUd 
Jn tkt 4mt mif 

DiagrtOfo^ • posslbto 
"discrimination'* 
problem. Falluro to 
dlscrlminato botwoan 
two sets of symbols 
which raqyira different 
treatment. 



M: ACTIONS 

-CCNEMllZATtONS 

*DISCNININATI0MS 
(an optional 
diagnosis) 



e.j., TtJT ITEM 

Voa mUk to mcA*a%e 
tkt magn^^tuU oi tkt 
buoywU iQKct ixtxtti 
on € submtA§td objtU. 

Ji mjit about 
mtMuKii^ it in tkt 
ioUmijn^ tm mifk. 
mutd you get tkt ^mt 
UAutJt 04 n diijitAS^ 
KtMuttf 

(f) McMttUiig tkt 
diiitJitnct in 
n^tifikt lifccii tkt 
objtct iA mijghtd 
ijn nix; 

AMD 

(2) HtJUWiins 
ifkxgiHJt oi tht 
owiAitM tkat 
f»jUuJbU iKom 
^ubmtAjqinq tkt 
objtcjt. 

Tkt mauuKtmjutJk tkojt 
KtAutt mitt be: 

l(iui\fattnt 

^on- iqwii/aJttnt 



Ne: IN^TS and ACTIONS 
-ASSOCIATIONS 



e.g., TEST ITPM 

GIVIH: RtpfLtAtntAtLon 
oijun atom 

-A 



Wkick oi tkt ioLLoiAiMq 
totntu poKticlu in 
an atom CAHAJtcJLbfl 

II) ?Hoton at A, 

tttJttfLon at B 

If) lUcJUion at A, 

p^aton at 8 

Diagnoses possible 
failure to associate 
the electron and 
proton labels (ACTION) 
with the appropriate 
location represented 
(INFUTS) 



STEP 



F.3 



COMPLETION CHECKLIST 



IDENTIFIED 


PERFORMED 


PRODUCED 


FORMS COMPLETED 


Need for 
supplementary 
diagnostic test 
i terns 


Planned use of 
criterion test 
items for diagnostic 
purposes 












Diagnostic probing 
questions for 
interviewing 
students 









Diagnostic test 








i tems 
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STEP F.4 



F.4 



Try out and revise testing procedures. 



F.4.1 



Administer testing procedures to a sample of the target audience 
as a means of assessing and upgrading the mechanics of the testing 
procedures. 



F.4.2 



Perform an analysis of the target audience by administering the 
revised tests. 





OVERVIEW 



INPUT 

TESTS 

-Criterion 

-Subcriterion 

-Diagnostic 

TESTING PROCEDURES 

-Diagnostic probes 



ACTION 

Administer to sample 
of target audience 
and revise as necessary 



OUTPUT 

Information about: 

-Adequacy of test 
mechanics 

-Target audience 



F,4.1 



Criterion tests, 
subcriterion tests, 
diagnostic tests, 
and probing procediires 



Tests with '^mechanics'* 
revised on basis of 
tryout results 



iv 



Administer to senple 
of target audience 
+ 

assess inadequacies 
in mechanics 
+ 

revise mechanics ii 



Administer to sample 
of target audience 



Revised and adequate 
mechanics of testing 



iii 



RESULTS INDICATING 

►Whether teste work 

-What are difficult 
learning problems 

"What may be omitted in 
developing instruction 




IA8 



STEP 



F.4 



F.4.1 



F.4. 2 
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PAGE INDEX 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 


-MATRIX: What 
features of 
test mechanics to 
assess . . I5't, 155 


-MATRI': How to 
assess mechanics 
of testing . . 156 


-MATRIX: Adequacy 
of assessment 
of testing 
procedures . . 159 


SUMM RY OF 
PROCEDUP.es . 158 




-MATRIX: Types 
of information 
revealed by 
tryout 

results . . 163-165 




-MATRIX: 
Assessing 
adequacy of 
test tryouts . 16S 


SUMMARY OF 
PROCEDURES . . 168 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A revised set of testing procedures which have been 
empirically tried out. 


WHAT YOU WILL 
WORK FROM 


(1) AM tests (crfterfon, subcr! ter ioOp dfagnostic). 

(2) Probing procedures. 


WHAT YOU WILL 
DO 


(1) Administer above to a sample of the target 
audience, 

(2) Assess fnadequycles In mechanics of testing pro- 
cedures (Instructions, statement of problems, re- 
cording of results, probing procedures, etc.). 

(3) Revise mechanics. 


FORMS YOU WILL 
USE 


None 
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INPUT 



Criterion tests , 
subcritcrion tests, 
diagnostic tests, 
and probing procedures 



DESCRIPTION OF Sub-STEP 



F.4.1 



ACTION 



Administer to sample 
of target audience 
+ 

assess inadequacies 
in mechanics 
+ 

revise mechanics 



11 



OUTPUT 



Revised and adequate 
mechanics of testing 



iii 



Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: VWiat 




-MATRIX: How to 


-MATRIX: Adequacy 


SUMMARY OF 




features of 




assess mechanics 


of assessment 


PROCEDURES . 


. 158 


test mechanics 


to 


of testing . . 156 


of testing 






assess . • 15^, 


155 




procedures . . 159 







Required MaleiKils 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Diagnostic probing 
procedures (carried 
forward from) 


F.3.3 


Cr i terion , 
subcritcrion, and 
diagnostic tests 
(carried forward 
from) 


F.3.3 
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Sub- STEP I p,4j 



INPUT 



Criterion tests, 
subcriterion tests, 
diagaostic tests, 

and probinc procedures 



JOB DIAGRAM 



ACTION 



Administer to sample 
of target au Uence 
+ 

assef^s inadequacies 
in mechanics 

revise mechanics 



ii 



OUTPUT 



Revised and adequate 
mechanics of testing 



iii 



Assessment of 
mechanics of: 

Tests 

-Diagnostic probing 
procedures 

i den t i f i es I nadequa c i es 



i.a 



Assessment of 
mechanics of: 

•Tests 

Diagnostic probing 
procedures 

identifies 
no inadequacies 



l.b 



Hake revisions in 
mechanics of testing 



ii.a 



No testing 



n.b 



Adequate: 

-Instructions 

-Statement of GIVEN 

-Display of 
INPUTS /OUTPUTS 

-Taking and recording 
of ACTIONS 

-Diagnostic probef^^i I 



Adequate: 

-Instructions 

-Statement of GIVEN 

-Display of 
INPUTS/OUTPUTS 

-Taking and recording 
of ACTIONS 



-Diagnostic probes 



1 1 Kb 
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BACKGROUND IIJFORf-lATION 





page 


Identification of verbal components 
of testing procedures which should 
be assessed 




Identification of functional 
properties of testing procedures 
which should be assessed 


155 


How to obtain information needed 
to assess adequacy of testing 
procedure;^ V"^ 


156 
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IDENTIFICATION 
MATRIX 



cRITtKIA FOK IDENTIFYING WHAT TYPES OF !MFOP>!AT!ON APE NEEDED IN 
ASSESSING THF ADFOllACY OF VERBAL COMPONENTS OF TESTING PROCEDURES 



TYPES OF 
INFORMATION NEEDED 



CRITERIA 



Are INSTRUCTIONS 
adequately understood 



Doee the student 
undcTBtand: 

-What INPUTS or INPUT properties 
to attend to or observe 

'What the task ia he is expected 
to perform; what ACTION (s) he is 
expected to exhibit 

'What he is expected to turn out 
(OirrpUTS) 



Are statements of TEST QUESTIONS 
adequately understood 



Does the strident 
understand the language 
used to describe: 

-The GIVEN in a test question 
or problem 

• •Description of INPUTS 

••Description of ACTION options 
(in miltiple choice tests) 

^-The queations asked in probing 
intervieu^s designed to produce 
diagnostic infomation 



SEE PAGE FOR WAYS TO MAKE THIS ASSESSMENT AND THE ASSESSMENT DESCRIBED 
ON THE OPPOSITE PAGE. 



ERIC 



F.4.1 

IDENTIFICATION 
MATRIX 



CRITERIA FOR IDEHTIFYING TYPES OF INFORMATION NEEDED 
TO ASSESS THE FUNCTIONAL ADEQUACY OF TESTING PROCEDURES 



TYPES OF 
INFORMATION NEEDED 


Are visual and/or verb.^J 
INPUTS adequately 
DISPLAYED 


Is there an adequate 
opportunity for the 
student to RESPOND, 
i .e. , take an ACTION 


Arc visual and/or verbal 
OUTPUTS adequately 
DISPLAYED^ 


CRITERIA 


la the student able 
cuiequately to: 

'See or read INPUTS 

'Hear INPUTS 

'Feel, taste, or smell 
INPUTS 


Is the student able 
adequately to: 

'Write or record his 
ACTION 

'Produce the required 
motor ACTIONS 


Is the student able 

adequately to: 

'See or read OUTPUTS 
'Hear OUTPUTS 

'Feel, taste, or smell 

OUTPUTS 



^Adequate display of OUTPUTS is particularly inportant vhen the OUTPUTS became 
the INPUTS for serial ACTIONS in a chain . 





INPUT P^p^Ktlu: 


ACTION PKop^txe^*^ 


OUTPUT VKoptKtieM 




-Lcjglbitity ok 


'Space, OK toaitioii^ to 
KtcoKd ACTIONS 


-Legibltitij ok 
vl^ibititi/ 


EXAMPLES 




• 'Space to voK4j:c 


'AudibiLitij 




-hicinipulabititif 
-CapabiZitij bcxng 


• • Space to c/iecfe 

Opti^OiU 

••AcceAA to ptact^ to 
point to 

-Oppontanlti^ to take 
motOK ACTIONS 

• •Eqtxipmcnt f^eAtoKus 

that vooKk iknob^ to 
tuKn, 6u)ttche^ to 
optAatt, etc. ) 


'ManiputahitxXi^ 

-Capab/JUJbj o^ beting 
toAted OK 6metZed 
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DETERMINING HOW TO OBTAIN INFORMATION NEEDED TO 
ASSESS THE ADEQUACY OF TESTING PROCEDURES 
(TWO WAYS WHICH CAN BE USED SEPARATELY OR TOGETHER) 



DECISION 
MATRIX 



ASSESSMENT 
CONDITIONS 


Our ( ng 
student PERFORMANCE 
on testing procedures 


Ouri ng 

INTERVIEWS with stTidents 
held during or following 
administration of 
test Items or test procedures 


ACTION 
TO TAKE 


'Make observations of what student 
does to determine if: 

-*He attende to relevant INPUTS 
input properties (rather 
than to iri'elevant ones) 

•*fie attenpts the required task 
rather than som0 other task 

•*He produces the required output 
type rather than something else 

'^iake a record of wJjcn a^id over 
what students: 

• • Indi cate confua ion 

Spontaneously ask questions 

•^Indicate an inability to moke 
out what the INPUTS ai'e 

Indicate problems or 
difficulties in taking ACTIONS 


-Solicit from students information 
about: 

••Their difficulties or confusions 

••What tJiey thought about 
instructions or tasks which 
were involved 

••Whether they can see or read 
INPUTS 

••Whether they had problems taking 
any of the required ACTIONS 

• •Whether they understood 
^'probing" questions 



o 
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JOB PROCEDURES 





page 


SUfWARY OF PROCEDURES 


158 


Standards for adequacy 
of assessment 


159 
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ILLUSTRATION SUKM^RIZINC PROCEDURES INVOLVED JN ASSESSING AND REVISING 
THE MECHANICS OF TESTING PROCEDURES* 



12 



Adm i n i s tc r tcs t s {criterion^ 
subcri terion^ or diagnostic) 
and/or administer diagnostic 
probing procedures to smaH 
sample or target audience: 

-Those already proficient 
and/or 

-Those who are to receive 
instruct ion 

Observe student performance 
during testing procedures 
to obtain information about 
deficiencies in the 
mechanics of testing 
procedures 

Hake a record during testing 
of spontaneous student 
indications of difficulties 

Following conclusion of 
testing procedures, 
interview students about 
dif f icul ties 



a. Kake revisions in testing 
procedures : 

- In instructions** 

-In statement of test 
problems or questions 

-In display of INPUTS 

-In provisions to allow 
ACTIONS to be taken 
and/or recorded 

-In display of resulting 
OUTPUTS 



^^See Section G (FORMULATION 
OF INSTRUCTIONAL STRATEGIES) 
for guidelines on 
'^inatruetions'* designed to 
control student attending 
or observing behaoior 



^The tryout and revision of testing procedures should be carried out 
more than once (when feasible to do so) until students perfom as 
required (e.g», attend to the proper INPUTS or attempt to perform 
the task required of them)* 
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F.4.1 

STANDARDS 
MATRIX 



CRITERIA fOR EVALUATING THE PROCESS OF ASSESSING AND REVISING 
THE MECHANICS INVOLVED IN TESTING PROCEDURES 



PROPERTIES 


COMPLETENESS 


VALIDITY 


RELf ABILirr 


OF ASSESSMENT/REVISION 


OF ASSESSMENT/REVISION 


OF ASSESSMENT/REVISION 




Cover a all 


'-Assessment is based 


'Assecsncnt is based 




of the following: 


an data: 


on data from a sample 




-Statenent of "CIVET' 


Observations 


(at a minimum, a 
half dozen people) of 




in teat problem 
-Display of INPUTS 

and /or recording 
ACTIONS 

-Display of INPUTS 
Cover3 the following 


Interviews 


the tartlet audience 


CRITERIA 


(Is NOT based on 
subjective judgments 
by you, the designer, 
as to adequacy of 
mechanics) 

'Assessment is based 
on data from membt^ra 
of the target audience 




when applicable: 








'Revision is a proceaa. 






'Diarrnoatic teat probes 


continuing through 






used in intervici^^a 


cycles until new 
asaeasment shous 
mechanics to be 
adequate 





ERIC 



J 53 



PREVIEW OF THE NEXT SubSTEP 



1 

YOUR PRODUCT 


An identification of: 

^''the adequacy of the test; 
'"^anticipated leami>ig probleme; 

^^what may be omitted from the instructional program. 


WHAT YOU WILL 
WORK FROM 


(1) Tests and revised testing procedures. 


WHAT YOU WILL 
DO 


(1) Administer tests to a sample of the target 
aud lence 

(2) Review results as a basis for determlnfng adequacy 
of tests and making decisions about the develop- 
ment of the program. 


FORMS YOU WILl 
USE 


None 
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DESCRIPTION OF Sub STEP 



INPUT 



Te^ts with "mechanics" 
revised on basis of 
tryout results 



IV 



F.4.2 



ACTION 



Aduinister to sa-niple 
of target audience 



OUTPUT 



RESULTS INDICATING 
-Whether tests work 

-What are difficult 
learning problems 

-What may be omitted iii 
developiTiji 5nsti*uction 

vi 



Job Aid Contents 

CRITERIA FOR 



IDENTIFYING INPUTS 


ACTION TO BETAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 


-MATRIX: Types 




-MATRIX: 


SUMMARY OF 


of information 




Assess ing 


PROCEDURES . . 168 


revealed by 




adequacy of 




tryout 




test tryouts . j^g 




results . . I63-I65 







Required Materials 



COMPLETED r.lATERiALS 

STEP 


coMPLFTPo For;r,iG 

STEP 


BLANK FORMS 






Tests with mechanics 
revised based on 
tryout results 

























































o 
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Sub- STEP 



F.4.2 



JOB DIAGRAM 



INPUT 



Tests with "mechanics" 
revised on basis of 
tryout results 



iv 



ACTION 



Administer to sample 
of target audience 



OUTPUT 



RESULTS INDICATING 
-Whether tests work 

-What are difficult 
learning problems 

-What may be omitted in 
developing instruction 




Results of tests 
administered to sample 
of BEGINNERS 



IV. a 



Analyze results 



v.a 



Ident f ication of 



-Whether BEGINNEkS can 
a I ready exhi bi t 
criterion behavior 

-Whether they have 
previous ly learned 
behaviors which can 
interfere wi th the 
learning of criterion 
behavior 

vi .a 



Results of tests 




Analyze results 


administered to sample 






of students who have 






COMPLETED existing 






i nst ruct ion 








► 




iv.b 




v.b 



-Whether COMPLETERS 
have residual 
difficulties with 
respect to the 
cri terion behavior 



vi.b 



Difference in results 
of tests administered 
to COMPLETERS and 
BEGINNERS 



iv. 



Analyze results 



v.c 



•Whether the test 
tests what It is 
supposed to 



BACKGROUND INFORMATION 





page 


Identifying potential samples 
from a target audience for tryout 
of test items 


\Gk 


Identifying types of information 
(and its uses) which tryout of 
tests provides 


165 
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F.4.2 

IDENTIFICATION 
MATRIX 



CidTERIA FOR IDENTIFYING TWO TYPES 
OF SAMPLES FROM THE SAME TARGET AUDIENCE 



TYPES OF 
SAMPLES 


Those who have 
already COMPLETED instruction 
on criterion behaviors 


Those who arc 
BEGINNERS 


CRITERIA 


-Those who have just satisfactorily 
coTjplcted an existing course or 
training program 

-Those who have been on a job (for 
some time) and are judged to be 
performing adequately 


-Those who arc about to begin a 
course or training program 

-Personnel just abdut ready to 
begin a job 




EXAMPi.ES 


e.g., htadojnts ^^ho have jiU^t 

completed a portion OK 
a paJiticjxtoK topic in a 
tlvUid gxadz matJi aouue. 

Students in both groups 
for general inteUigenc 

eg*, tQjOLoiizU on thz job 


z.g., 6tadc\itb toho o/ie about to 
b^gin a po^tJjon of^ ok a 
poKticiitaA topic in 
g\adc matli couJtbc 

are roughly matched 
:e or abM i ty 

e.g., h^imiing tmdizA tnjouinzzJi^ 
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CR(TC«fA rOR iOCfinrytfX, TrPCS or {UfCRnAHOU to BC OCRfVCO fRON 
Tf^YOuT RfSULTS CN TESTS WITH TWO TYPES OF SAMPLE FROM TARGET AUDIENCE 



Nlif ICAIION 



TYPCS Of 


Results for 6CCINNCRS 
on tests: 

• 'Cr i ter ion trs ts 

• 'Subcr 1 ter fon tests 


P'^sulls for 
COURSE COMPLETERS^ 
on tests: 

••C.lterion tests 

• -Subcri terion tests 

••Diagnostic tests 

• •D f agnost 1 c probi nq 
procedures 


OIPFERENCE 
in results for 
COURSE COMPLETERS 
and BEGiNNERS 
on tests: 

••Criterion tests 

• -Subcr i ter ion tests 


CRITERIA 


Pcsults idertifu: 


Reeulta identify: 


Heeults idcntifi^: 


already have learned: 

' ' The criteHon 
b*ihai?ior 

•'Parte of the 
criterion behavior 

'Whether the bc(jinnere 
have previous ly 
learned: 

* * Behui^ i.ore which may 
interfere hfith the 
teaming of the 
criterion behavior 


-Whether carplctere 
hcn>e rceidual 
difficulties with: 

"The criterion 
behavior 

"Parte of it 

"Canponent ekille 


^Whether the test 
adequately teste foi 
proficiency at: 

"The criterion 
behavior 

"Parte of it 


HOW THE 
iNrORMATiOM 
CAM BE USED 


-C».i Influence the 
decision of whether 
or not to deve lop 
i nstruct ion«l materials 
for a part icul ar 
criterion behavior (or 
part of It) 

••ff all beginners can 
exhibi t a particular 
criterion behavior, 
no Instruction on 1 t 
need be prepared 

-Can influence the 
decision of whether or 
not to a Mow some stu- 
dents to bypass 
Instructional matertaH 
covering that criterion 
behavior 

•■If some students can 
exhibit the criterion 
behavior and some 
cannot, Jn acconm- 
dating individual 
dffferences, you ma^ 
plan to allow those 
who can to bypass 
the related tnstruc 
tionat materials 
(resulting in more 
efficient use of 
instructional time) 

-Can influence formu- 
lation of instruc- 
tional strategies to 
overcome potential 
interference exist- 
ing behaviors may 
have wl th the learn- 
i ng of new ones 

e.g. , old associ- 
ations which 
may interfere 
wl th the 
learning of 
new ones | 


-Can feed Into the 
strategy formulation 
(See Section C) so 
that instruction on 
the weak area can be 
di rected to specific 
learning problems 

* Corrptetion of 
exieting coureea^ 
not the one you are 
planning to develop^ 


-Can be used to revise 
test 1 terns which do 
not adequately measure 
what they are intended 
to measure 

e.g., paper-and'penci t 
items %^lch are 
badly con- 
structed may 
lead to no test 
dl f ferences • 
tven when 
beginners have 
not previously 
}eerne6 the 
cri terlon 
behavior 
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JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 


)68 


Standards for adequacy of tryout 
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F.4.2 

STANDARDS 
MATRIX 



CRITERIA FOR ASSESSING THE ADEQUACY 
OF TRYOUT OF TESTS AND TEST PROCEDURES 



PROPERTIES 



CRITERIA 



SAHPLING 
OF TARGET AUDIENCE 



'Target andimnoe r^prms^ntrnd by 

both: 

• •Beginner 8 

••Completere (of previoue ooureee 
or program) 

'Eaah Ufpe of ecmrple includes a 
minimen of ten people 



COHPLETENESS 
OF ANALYSIS OF TEST RESULTS 



'Analyeie atten^pta to identify: 

••Reeidual diffioultiee oompletere 
have (ae a basie for foeuevng 
on the probleme (hiring fomula^ 
tion of inetruetional 
etrategiee) 

••fcTiat begimere already oan do 
(as a baave for individualizing 
eeheduling) 

••What beginnere already do (as a 
baeie for detemining i>hat may 
interfere with what hoe to be 
learned) 
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STtf I F.4 



COMPLETION CHECKLIST 



IDENTIFIED 


^RFORMED 


PRODUCED 


FORMS COMTLETED 




Assessed adequacy 
of mechanics of 
testing procedures 


A revised set of 
tests and/or 
diagnostic probing 
procedu res 






-Pottntial learning 
problems 

-What t«r9«t audienci 

(beginners) already 
knoM 


Tried out tests 
with sample of 
target audience 


A revised set of 
tests 
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